Design Is Refactoring

Presentations

2022-12-21T00:00:00+00:00

Presentations

I’ve given a lot of presentations over the years and it’s something I really enjoy doing. I realized that I’d shared some of these, but not all of them. I thought it would be nice to gather them all in one post.

There are other talks that aren’t here, either because they weren’t recorded or those videos have been lost to time.

Introduction to Kafka Connect

August, 2022

A introduction to and demo of Kafka Connect given to the UMN IDEAA group.

Slides
Video

Hiring with CARE

October, 2021

Designing and using principles to improve the tech hiring process.

Slides
Video

People over Process, Feelings over Hype

May, 2020

My opinions on agile, therapy and how to deal with the unexpected.

Slides
Video

External Data and Events

October, 2019

A tour through the various ways I and my team have dealt with external data sources. As well as the differences between external data and external events. ETL, Live Querying, Database Links, Kafka, CDC, and more.

Slides
Video

Break Silos with Github

November, 2017

Using the tools offered by GitHub to break technical silos.

Slides
Video

Student Degree Progress Service

June, 2017

An tour of getting a student’s degree progress data from UMN systems.

Slides
Video

Even Hemingway Wasn’t Hemingway

September, 2015

Given at Rocky Mountain Ruby, a talk about the value of rewriting code.

Slides
Video

Introduction to Kafka Connect

2022-08-01T00:00:00+00:00

A introduction to and demo of Kafka Connect given to the UMN IDEAA group.

Hiring with CARE

2021-10-01T00:00:00+00:00

Designing and using principles to improve the tech hiring process.

People over Process, Feelings over Hype

2020-05-01T00:00:00+00:00

My opinions on agile, therapy and how to deal with the unexpected.

External Data and Events

2019-10-01T00:00:00+00:00

Break Silos with Github

2017-11-01T00:00:00+00:00

Using the tools offered by GitHub to break technical silos.

Student Degree Progress Service

2017-06-01T00:00:00+00:00

A tour of getting a student’s degree progress data from UMN systems.

Combining Reek and Overcommit

2017-04-05T22:29:01+00:00

My team has been using Overcommit for a while. We like how easy it makes it to set up Git commit hooks for the entire team and how easy it is to see what those hooks are.

Recently we decided to add Reek to our Code Quality process. To make it automatic we wanted to run Reek as part of our pre-commit hook. Overcommit already has a default hook for Reek, so making Reek run as part of every commit was super easy. Great!

But.

When committing Reek warned us about smells that we couldn’t change. For example, Reek doesn’t like the default Rails migration syntax for a variety of (good) reasons. But my team isn’t going to change the Rails migration syntax so having Reek warn us about those changes wasn’t useful.

Reek offers many ways to silence these warnings. A recent addition is the exclude_paths configuration option. If you have this in your Reek config file

# Directories below will not be scanned at all
exclude_paths:
  - db/migrate

Then when you run reek . it won’t tell you about any smelly files in db/migrate. Hooray!

But.

When we ran Reek in our pre-commit hook it kept warning us about files in db/migrate. Why?

Well, what Overcommit does is ask Git for a list of all files that have changed. It then passes each of those files to Reek. So, if a file changes in db/migrate then Reek is run with

reek ./db/migrate/file_that_changed.rb

And Reek’s behavior is to always check a file if given a path to a specific file. Even if the file is in an excluded directory.

Sigh.

After checking if it was cool with the Reek team we submitted a couple of PRs that gave Reek a new option --force-exclusion. We chose the flag name because it’s the same one used by Rubocop. Might as well be consistent. The flag tells Reek to ignore files in an excluded path, even when provided the path to a specific file. This work has been merged into Reek as of version 4.6.0, though you probably want 4.6.1 because of a bug with absolute paths.

For Overcommit to use the new flag you’ll need to update your .overcommit.yml file:

PreCommit:
  Reek:
    enabled: true
    flags: ['--single-line', '--no-color', '--force-exclusion']

If you skip that part Reek will not use the new flag and you’ll still get warnings about files in excluded paths.

Now you can easily combine the great work of the Reek and Overcommit teams. Thanks to them for managing such helpful projects!

Refactoring Rust: Primitive Obsession

2016-08-31T20:55:51+00:00

Thanks to their small, focused nature problems in Exercism tend to focus on language primitives. Transform this string into a different string, manipulate this number, etc. In the bob exercise students return a string based upon a string input. And in leap students return a boolean based on an integer input.

Students tend to rely on primitives when solving these problems. The example solutions for bob and for leap are indicative. They accept primitive input, use primitives to solve the problem and return primitive results.

This is, in short, primitive obsession. And for people just starting to learn Rust (or any language) primitive obsession is fine. Beginners on the Dreyfus model aren’t yet modeling domains or complex concepts; they are still learning the techniques these problems exercise: syntax, primitives, control flow, etc.

But as students progress through the exercises their skills will evolve and they’ll be able ponder maintainability and design. The question will arise – how do I write good Rust code?

I certainly don’t have the answers to that question, and in the Rust track we’re barely starting to think about how to help students answer that question. Should exercises actively discourage certain code smells, like Primitive Obsession? What would that discouragement even look like? But the discussion has led me to start thinking about what code smells look like in Rust and how we can apply standard Refactoring patterns to remove them.

As always, this is easier to understand with an example. Let’s look at Primitive Obsession in one of the most recent additions to the Rust exercise track: Bracket Push.

The problem is straight-forward: return true if the brackets in a string are balanced (all open brackets are closed in the correct order), and return false if they are not. There are a few different ways of solving it; I started with the following:

pub fn are_balanced(input: String) -> bool {
    let mut unmatched_brackets: Vec<char> = Vec::new();

    for current_bracket in input.chars() {
        if let Some(most_recent_unmatched) = unmatched_brackets.pop() {
            let is_matched = match (most_recent_unmatched, current_bracket) {
                ('[', ']') => true,
                ('{', '}') => true,
                ('(', ')') => true,
                _ => false,
            };
            if !is_matched {
                unmatched_brackets.push(most_recent_unmatched);
                unmatched_brackets.push(current_bracket);
            }
        } else {
            unmatched_brackets.push(current_bracket);
        }
    }

    unmatched_brackets.is_empty()
}

fn main() {
    assert!(are_balanced("{[]}".to_string()));
    assert!(!are_balanced("[{]}".to_string()));
}

There are better ways to structure the logic, sure. But set that aside for a minute and let’s focus on the primitives. In the above code I count five of them:

String*
char
boolean
Vector*
Tuple

* The Rust book doesn’t list these in the Primitive types section, but I’m fine with thinking of them as primitives. It’s a fuzzy line.

The are_balanced function must accept a string and return a boolean to satisfy the tests. But inside the are_balanced function we can do whatever we want, and yet what I chose to do was use a bunch of primitives.

The primitives inside the function have satisfied all of the criteria of the Primitive Obsession smell: they were easy for me to use and they passed the tests. However, they obscured my code’s intent and made it hard to change. I tried to provide some useful variable names but I still find that I have to read the entire function, slowly, before I understand what it’s doing. This is not code that is easy to understand; and poorly understood code is rarely well maintained code.

So, let’s fix it! Checking with Martin Fowler the official fix for Primitive Obsession is to replace the primitive with an object. Well, Rust doesn’t have objects so we can’t do that. But Rust still gives us all the tools we need to refactor away these primitives – Structs and Traits will do most of the heavy lifting. If we replace our primitives with structs and traits that model the domain we will improve our code’s clarity and maintainability. Where to start? Since the first primitive our function deals with is the String, let’s start there.

We’re not going to change the test, because refactoring should not change the behavior of existing code. Our function will still accept a string, but we can convert it to domain-specific Struct inside the function.

What should we call this Struct? Naming is hard, as always. Since it contains a bunch of brackets let’s just call it Brackets for now. Maybe a better name will occur to us later. Brackets will start off containing our input string.

struct Brackets {
  input: String,
};

How do we create our Brackets inside are_balanced?. We could do this:

pub fn are_balanced(input: String) -> bool {
  let brackets = Brackets { input: input };
  //...

And that’s…OK. But it’s not idiomatic. We’re trying to convert a String to a Brackets and Rust has a standard way to convert types – the From trait. You may have seen this trait before in the commonly used String::from(....) function. But you can implement From for any type!

struct Brackets {
  input: String,
};

impl From<String> for Brackets {
  fn from(input: String) -> Self {
    Brackets { input: input }
  }
}

Wait! That’s pretty much exactly the same thing we did in the last example. What’s the benefit of implementing From? First, we’re both following a Rust idiom and establishing a example for our code. And this example will guide us if (read: when) we want to create a Brackets from something else? A vector? A &str? Time will tell, but we’ve already established the protocol to follow whatever happens.

All of this is good…but it doesn’t help us. Our code still relies on the behavior of a string primitive:

pub fn are_balanced(input: String) -> bool {
    //..
    for current_bracket in input.chars() {
    //..
    }
}

Our Brackets doesn’t implement chars(). We can make it do so:

use std::str::Chars;
impl Brackets {
  fn brackets(&self) -> Chars {
    self.input.chars()
  }
}

That feels unsound to me, though I can’t put a finger on why. I suspect this shortcut will come back and bite us later. Let’s be a little more thorough and idiomatic.

chars returns an iterator and – just like the From trait and type conversion – Rust has a standard way of adding iterators to your Structs, IntoIterator

impl IntoIterator for Brackets {
  type Item = char;
  type IntoIter = ::std::vec::IntoIter<char>;

  fn into_iter(self) -> Self::IntoIter {
    self.input.chars().collect::<Vec<char>>().into_iter()
  }
}
// Note: I mostly copied this code from The Book; there may be a better way of implementing this Trait for Brackets.
// Update: Ms2ger showed me how to implement this without a vector:
//   https://github.com/IanWhitney/designisrefactoring/pull/22#discussion_r77125688

With this done, we can use Brackets throughout are_balanced

struct Brackets {
    input: String,
}

impl From<String> for Brackets {
    fn from(input: String) -> Self {
        Brackets { input: input }
    }
}

impl IntoIterator for Brackets {
    type Item = char;
    type IntoIter = ::std::vec::IntoIter<char>;

    fn into_iter(self) -> Self::IntoIter {
        self.input.chars().collect::<Vec<char>>().into_iter()
    }
}

pub fn are_balanced(input: String) -> bool {
    let brackets = Brackets::from(input);
    let mut unmatched_brackets: Vec<char> = Vec::new();

    for current_bracket in brackets.into_iter() {
        if let Some(most_recent_unmatched) = unmatched_brackets.pop() {
            let is_matched = match (most_recent_unmatched, current_bracket) {
                ('[', ']') => true,
                ('{', '}') => true,
                ('(', ')') => true,
                _ => false,
            };
            if !is_matched {
                unmatched_brackets.push(most_recent_unmatched);
                unmatched_brackets.push(current_bracket);
            }
        } else {
            unmatched_brackets.push(current_bracket);
        }
    }

    unmatched_brackets.is_empty()
}

fn main() {
    assert!(are_balanced("{[]}".to_string()));
    assert!(!are_balanced("[{]}".to_string()));
}

By implementing a Struct and two traits we’ve:

Given a name to our previously-nameless String
Introduced an idiomatic way of creating our Brackets
Used idiomatic Rust to show that Brackets can be iterated
Made our code longer

No doubt about it, our code is longer. But it also more clearly illustrates what it’s doing, and I’m OK with that tradeoff.

That’s enough for now. Of our original list of five primitives there remain 3 that I would like to remove. Next time we’ll tackle the Tuple.

If you want to comment, question, complain, &c. then you can do so on twitter, at ian@ianwhitney.com, or by leaving comments on the pull request for this post.

The Lifetime of Food

2016-08-10T21:16:00+00:00

Imagine you are out to dinner with some friends. You are all stereotypical Millennials so photographing your meal is expected. For every dish you order each of your friends borrows it, takes a photograph, and then returns it to you.

Well, that’s the plan anyway. Some of your friends are not the most reliable. Will they always return the right dish to you? Will they try to sneak a bite? Human friends are so unreliable. But if your friends were Rust functions you’d have nothing to worry about, thanks to Rust’s rules around borrowing and lifetimes.

Let’s model this dinner party in Rust and see if we can learn a thing or two about borrowing and lifetimes.

We’ll start off with the basics, a Dish that can be photographed.

struct Dish {
    name: String,
}

impl Dish {
    fn photograph(&self, photographer: &str) {
        println!("Snap! {} photographed by {}", self.name, photographer)
    }
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    doro.photograph("Me");
    bruschetta.photograph("Me");
}

We have a Dish struct which contains a name. There’s one function on Dish, photograph. You get your dishes from the hot new Italian/Ethiopian fusion restaurant and take their pictures.

Let’s introduce one of your friends. We’ll start off with your most honest and trustworthy companion. Being honest this friend wants to borrow a dish photograph it and return it. They don’t want you giving them more than one dish – that could get confusing! When converted to a Rust function, this friend looks like:

fn honest_friend(dish: &Dish) {
  dish.photograph("Honest Friend")
}

Our dinner-party now looks like:

struct Dish {
    name: String,
}

impl Dish {
    fn photograph(&self, photographer: &str) {
        println!("Snap! {} photographed by {}", self.name, photographer)
    }
}

fn honest_friend(dish: &Dish) {
    dish.photograph("Honest Friend");
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    honest_friend(&doro);
    honest_friend(&bruschetta);

    doro.photograph("Me");
    bruschetta.photograph("Me");
}

Note that your friend is not explicitly returning the Dish at the end of their function. Thanks to Rust’s borrowing rules they don’t have to. When your friend’s scope ends, the borrow ends.

fn honest_friend(dish: &Dish) {      //-- Borrow of the dish begins here
  dish.photograph("Honest Friend")   // |
}                                   //-- Borrow of the dish ends here

A new friend joins your table. This friend likes to play pranks, but they don’t always go as expected. Your Trickster Friend says, “Loan me a dish and I’ll give you a dish back. Trust me, I’m going to return to you the same dish you loaned me.” You have your doubts.

fn trickster_friend(dish: &Dish) -> &Dish {
    dish.photograph("Incompenent Friend");

    // Tricks and Pranks ensue

    // Is this really the same dish you loaned?
    mystery_dish
}

struct Dish {
    name: String,
}

impl Dish {
    fn photograph(&self, photographer: &str) {
        println!("Snap! {} photographed by {}", self.name, photographer)
    }
}

fn trickster_friend(dish: &Dish) -> &Dish {
    dish.photograph("Trickster Friend");

    // Your friend is either returning the same dish or a different one.
    // Try running both options by uncommenting the assignments of mystery_dish

    // Option One: Returning a different one
    // let mystery_dish = &Dish { name: String::from("Pad Thai") };

    // Option Two: Returning the same one
    // let mystery_dish = dish;

    mystery_dish
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    let mystery_dish = trickster_friend(&doro); 

    mystery_dish.photograph("Me");
    bruschetta.photograph("Me");
}

Try both options and you’ll see that your trickster friend can not prank you by giving you Pad Thai. The Pad Thai value is owned inside trickster_friend and is destroyed when trickster_friend ends. This is a bit easier to see if you inline the trickster_friend function into your main function

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };

    let mystery_meal = {                                          //-- Mystery Meal is created here 
      let other_dish = &Dish { name: String::from("Pad Thai") };  // |  //-- Pad Thai is created here
      other_dish                                                  // |     |
    }                                                             // |  //-- Pad Thai is destroyed here
                                                                  // |
    me(mystery_meal)                                              //-- Mystery meal lives for longer than Pad Thai,
                                                                  //   so it can't contain a reference to Pad Thai. 
                                                                  //   Compiler Error
}

Our trickster friend shows us that if a function is going to receive and return a single borrowed reference it has to return the same reference it received. Even though our friend’s tricky plan failed we learned something from it!

A third, very impatient friend arrives to our party. They don’t have time for this “one plate at a time” nonsense; this friend wants you to loan them two dishes at once:

fn impatient_friend(first_dish: &Dish, second_dish: &Dish) {
    first_dish.photograph("Impatient Friend");
    second_dish.photograph("Impatient Friend");
}

struct Dish {
    name: String,
}

impl Dish {
    fn photograph(&self, photographer: &str) {
        println!("Snap! {} photographed by {}", self.name, photographer)
    }
}

fn impatient_friend(first_dish: &Dish, second_dish: &Dish) {
    first_dish.photograph("Impatient Friend");
    second_dish.photograph("Impatient Friend");
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    impatient_friend(&doro, &bruschetta);

    doro.photograph("Me");
    bruschetta.photograph("Me");
}

Just as with your honest friend, this works fine. We loan our two dishes to our impatient friend who takes their photos. When impatient_friend ends, so do the borrows.

But your trickster friend sees this exchange and gets an idea. They now know that they can’t return a plate of Pad Thai, but they figure that if they borrow 2 plates and only return one, they can keep that extra plate for themselves.

struct Dish {
    name: String,
}

impl Dish {
    fn photograph(&self, photographer: &str) {
        println!("Snap! {} photographed by {}", self.name, photographer)
    }
}

fn trickster_friend_again(first_dish: &Dish, second_dish: &Dish) -> &Dish {
    first_dish.photograph("Trickster Friend");
    second_dish.photograph("Trickster Friend");

    first_dish
    //ha hah hah, I get to keep the second dish!
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    let mystery_meal = trickster_friend_again(&doro, &bruschetta);

    mystery_meal.photograph("Me!");
}

Foiled again! If you run that you’ll get error: missing lifetime specifier.

Remember what we first learned from our Trickster Friend? If a function receives and returns a single reference then it must be the same reference, otherwise you could return something that does not live long enough. That’s easy for the Rust compiler to check as there’s only one option. But this function receives two references and returns one. How does the compiler ensure that the reference you return points to a value that lives long enough?

In this particular code both our values – doro and bruschetta – live long enough. So it can be hard to see why the Rust compiler is confused. But what if we changed our code to:

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    let mystery_meal = {
      let herring = Dish { name: String::from("Pickled Herring") };   //-- Herring created here
      trickster_friend_again(&herring, &bruschetta)                   // |
    }                                                                 //-- Herring destroyed here


    mystery_meal.photograph("Me!");
}

If trickster_friend_again returns bruschetta then that code would be fine. But if it returns herring we’d try to photograph a dish that no longer existed. This is unsafe, so Rust won’t let us do it.

If the Rust compiler ever has any questions about what reference a function will return then that function must explicitly declare a lifetime. Our trickster friend could declare one of four lifetimes:

Option One:

fn trickster_friend_option_one<'a>(first_dish: &'a Dish, second_dish: &Dish) -> &'a Dish

Option Two:

fn trickster_friend_option_two<'a>(first_dish: &Dish, second_dish: &'a Dish) -> &'a Dish

Option Three:

fn trickster_friend_option_three<'a>(first_dish: &'a Dish, second_dish: &'a Dish) -> &'a Dish

Option Four:

fn trickster_friend_option_four<'a, 'b>(first_dish: &'a Dish, second_dish: &'b Dish) -> &'a Dish

Let’s see the differences between those options:

struct Dish {
    name: String,
}

impl Dish {
    fn photograph(&self, photographer: &str) {
        println!("Snap! {} photographed by {}", self.name, photographer)
    }
}

fn trickster_friend_lifetime_one<'a>(first_dish: &'a Dish, second_dish: &Dish) -> &'a Dish {
    first_dish.photograph("Trickster Friend");
    second_dish.photograph("Trickster Friend");

    //Uncomment below to try to return second_dish
    //second_dish

    //Comment this out when you try to return second_dish
    first_dish
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    let mystery_meal = trickster_friend_lifetime_one(&doro, &bruschetta);

    mystery_meal.photograph("Me!");
}

In this example we’ve declared one lifetime, given it to the reference passed to first_dish and our return value. Trickster friend now must return first_dish.

Let’s try the herring code again:

fn trickster_friend_lifetime_one<'a>(first_dish: &'a Dish, second_dish: &Dish) -> &'a Dish {
    first_dish
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    let mystery_meal = {
      let herring = Dish { name: String::from("Pickled Herring") };   //-- Herring created here
      trickster_friend_lifetime_one(&doro, &herring)                  // |  this will always return doro
    }                                                                 //-- Herring destroyed here


    mystery_meal.photograph("Me!");
}

Using our current Trickster Friend this code will compile, because there is no way that mystery_meal will ever be herring.

Option Two is the inverse of Option One. The function must now return the second_dish. If we pass in herring as the first dish, our code still runs.

fn trickster_friend_option_two<'a>(first_dish: &Dish, second_dish: &'a Dish) -> &'a Dish {
  second_dish
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    let mystery_meal = {
      let herring = Dish { name: String::from("Pickled Herring") };   //-- Herring created here
      trickster_friend_lifetime_two(&herring, &doro)                  // |   this will always return &doro
    }                                                                 //-- Herring destroyed here


    mystery_meal.photograph("Me!");
}

Option Three gives the same lifetime to both parameters so the function could return either of them. Our herring code will fail to compile regardless of parameter order because mystery_meal could be bound to a destroyed value.

fn trickster_friend_option_three<'a>(first_dish: &'a Dish, second_dish: &'a Dish) -> &'a Dish {
  // logic to determine if first_dish or second_dish is returned
  dish_to_return
}

fn main() {
    let doro = Dish { name: String::from("Doro Wat") };
    let bruschetta = Dish { name: String::from("Bruschetta") };

    let mystery_meal = {
      let herring = Dish { name: String::from("Pickled Herring") };   //-- Herring created here
      trickster_friend_lifetime_three(&herring, &doro)                // |   this could return doro or herring
    }                                                                 //-- Herring destroyed here


    mystery_meal.photograph("Me!");
}

In our dinner party Option Four is functionally the same as Option One. We’ve declared two lifetimes but we’re never using the second one making it irrelevant. I’m sure there are times when having two lifetimes on a function is useful, but I haven’t come across them. The book says you can have multiple lifetimes, but don’t explain why. I’d love to know more about this.

Is there more to lifetimes than this? Almost certainly. But I hope that this analogy helps you grasp the fundamentals of why Rust has lifetimes. If you want to learn more about fundamentals of Rust, maybe read one of my earlier posts, Rust via its Core Values. I also suggest the New Rustacean podcast, which has helped me understand quite a lot about Rust.

If you want to comment, question, complain, &c. then you can do so on twitter, at ian@ianwhitney.com, or by leaving comments on the pull request for this post.