Big Data: How I Learned to Organize Apps, Files, and Folders Spanning Multiple Drives and Lived to Tell About It

Data organization can be a huge pain, but the effort expended to properly sort and organize your files and folders can be well worth it.

Over the years, data accumulates. In some form or another, I have been accruing over 14 years of data on my current computing setup, most of which was copied over when transitioning to a newer machine. Like most people, I never stopped to consider how to effectively utilize my multiple hard drives for organizing my data. I would collect and “organize” my files and folders “on the fly” – merely adding to the existing setup that I’ve carried over the years. My data was scattered across 5 volume partitions without a real clear plan in mind. In short, my “system” was a reactive response to what I already had in place 14 years ago, as opposed to thinking about how I use my data now.

Thanks to my iPhone, I take a lot of photos; as of this writing my collection spans 120GB. I’ve also accrued over 400GB of Photoshop related files. There are some files I need on a regular basis, some files I need on an occasional basis, and some files which I know I will never really go back and revisit.

I needed to make sense of what I used and what I didn’t need. I’m not the type to throw away GB upon GB, so I had to figure out a method by which I could still keep “old” data around, should I need it.

NOTE: Before undertaking such a task, I strongly recommend having a good data back-up system in place.

The first task on the docket was to make sure ALL my data backups worked. I have three separate approaches in my back-up strategy:

1. Time Machine for incremental back-ups (versioning)
2. drive cloning, via SuperDuper
3. Offsite cloud backup via Backblaze

After back-ups were verified to be in place, I began to pare down my Applications folder. Apps and games that hadn’t been used were uninstalled using CleanMyMac2. Multiple versions of the Adobe Suite (CS4, CS5 and CS6) were eliminated altogether.

After the application purge, I turned my attention towards reallocating space on my hard drives. My goal was to maximize efficiency on my hard drives – to utilize them to their fullest, with room to expand as needed. I took a sheet of paper and broke down the partition maps and determined clearly how I used my data.

I came up with 4 distinct categories:

• Art and Design: Brush Settings, Textures, Swatches, and assorted files related to producing my work, including any reference eBooks and the like. Design files that I’ve created that are two years or older also live here.
• Data: data I use on a regular basis, which includes documents, my comics, AND my three cloud-backups: Dropbox, Copy and OneDrive
• Photos: Contains family photos and videos.
• Music: Contains music, TV shows and other media that I consume on a regular basis

I have a boot disk, which contains the OS and all my apps. The boot disk, and the drives listed above, each have a dedicated back-up partition. Back-ups are cloned from the source drives using SuperDuper.

This is my system.

Each aspect of what I do lives on its own partition. Each partition has its own back-up partition and back-up schedule. Instead of digging through nested folder after folder, I’ve reorganized my files such that it is very easy for me to quickly access the files I need.

In addition to my back-up partitions, I have Backblaze configured to back-up all my source partitions. Time Machine is employed to make versioned back-ups of ONLY the Data partition – which is where all my current works live. The other partitions do not need Time Machine’s versioning feature.

The whole process of migrating data took two days. It was a pain to watch the progress bars crawl as gigabytes upon gigabytes were moved from one drive off to another. Most of my copying took place at night, so I could let the task complete while I slept. In the end, it was worth it. I now have a clear data system in place that works well for how I use my computer.

What special considerations do you take into account when organizing your data? What tools do you use to manage your data and back-ups? Share them with me in the comments below.

-Krishna