I’m working on a project right now that requires me to scrape a lot of various financial data from the SEC website. I started writing a bit about the process here. I’ll be adding more to that series soon.

One of the things that we originally wanted to build was a full list of companies with their CUSIP numbers. CUSIP numbers are essentially unique identifiers you can use to look up stocks (and other financial instruments).

After a fair bit of research, it doesn’t seem there is an easy way to get a full list of companies with their CUSIP.

This is a good example of a point in a software project where things can go in very different directions.

On the one hand, we could have accepted the feature at face value and spent the next week trying to cobble together a list of our own. Maybe it works out; maybe it doesn’t.

On the other hand, we could step back, look at the problem, and ask what we are really trying to solve. How could we think differently about this? Why are we doing it in the first place?

Here’s what we realized. Whenever we pull back 13F filings from the web, we get the CUSIP number and name of the company for each line item in the filing. Why not just use that to create our list of companies?

If we come across a company name and CUSIP we don’t have, just add it to the database. Simple.

What’s the drawback?

In some sense, the drawback is that we don’t have a “full” list of companies. On the other hand, none of the features that we are building really require a full list. We just need to know what companies exist in our dataset.

Perfect.

Recognizing these decision points isn’t always automatic. It takes a while to develop a sense for when you are sliding off the rails. I’ve personally witnessed a number of development projects go deep into the weeds on problems they really didn’t care about solving, and they didn’t think twice about it.

There’s not a guaranteed method that I have found to get this right 100% of the time. However, there are a few things that can help:

  1. If you’re an engineer, realize that it’s your responsibility to be skeptical and question feature ideas. Always ask if an idea being presented is the easiest, clearest way to do something.
  2. If you work in management, make sure to keep an open line of communication with the engineering team, and push them to question your ideas and their own ideas.
  3. Don’t buy into sunk cost fallacies. If you start down a road and realize it’s not optimal, don’t be afraid to turn back.

Need Help?

We are a full service software engineering firm. We can help you with everything from design to project management to development.

Got a project you’d like to talk about? Get in touch at info@centralstandard.tech.