Removing duplicate files and folders - Cleaning up my hard disks

Ever since I started using a computer about 20 years ago, I've carefully collected all the files I thought I might need in the future. Quite frankly, I absolutely needed none of the files I created before 10 years ago. Back then, I hoped that job interviewers might ask me to show the work I had done, but they never did. Having had the chance to see the HR processes from inside organizations in a few of my previous jobs, I can tell you that one should be lucky if an employer actually checks whether an applicant has done

 something in the past and what they have done. All these new fancy HR processes make people too busy having to complete too many interviews for open positions. For reporting's sake, they don't have the time to actually go and check an applicant's past projects and the quality of their work.

My habit of collecting and saving all the files I make for projects has gotten to a point where I need multiple high-capacity hard disks and cloud space. I save my project files, important tutorials, camera raw files, picture/video edits, templates, songs I like, movies I like, and many other files, including automated backups every three months. My digital hoarding disorder has gotten out of proportion to a level where I have a backup of backups on Google clouds and important documents on OneDrive.

So now, I have about 1 TB of cloud space, 8 TB of local external storage options on multiple devices, and a bunch of SD cards and pen drives that could total up to 500 gigs. Whoever reads this blog post five or ten years from now would smirk at these numbers, just as we now laugh at the 20 gig HD on our old P4 computer (Millennium Super computers: 😛), but for present-day specs, I think as an individual PC user trying to keep this much data is crazy. At the place I worked for in 2005, the whole organization had about 2 TB of space to manage their data.

In recent months, most of my weekends have been wasted in efforts to organize these files and eliminate duplicate files and folders. Within a few hours of starting this task, I figured out that it's not humanly possible to go through these thousands of folders and files one by one and delete the duplicate files. I wasn't ready to trust the exe HD clean-up utilities on the internet either. So, with a bit of Googling, I wrote up a bat code that could loop through the file list and identify the duplicate files and print the details on the cmd screen. Again, it was difficult to go through one by one and delete, and also dangerous to let the program automatically delete all at once.

After a few more attempts to modify the bat file code to selectively delete the files and folders by prompting the user, I found the code to be messy, complex, and not very user-friendly for the user to go through one by one again. By having the same bat logic as the core, I wrote a Java program with a few classes, achieving much better usability. I struggled a lot to fix the GUI freeze when the thread is busy problem. When java.io is busy in the current thread reading the files or in a separate thread, the GUI becomes non-responsive, and we cannot see the progress of the process. After drilling Stack Overflow with all I had, I found out that SwingWorker  is the answer to solve the problem. From there, it was easy peasy. Within a few hours, the "JJCleanFF" :) utility was ready, and I was able to delete a lot of duplicate folders and files.

For now, I'll share the jar file and instructions to use the utility with you. After cleaning up the code a little bit and adding documentation, I'll post it on GitHub and share the link in this post for the source code. So,

1. Download the executable jar file by clicking the below icon or visiting https://www.jeyaramj.com/downloads/JJCleanFF.jar

2. Open the downloaded jar file.

3. In the main window, click on "Select a folder" to select the parent folder in which the folders and files will be checked for duplicates

4. Once the folder is selected, there will be a message prompt for you to choose the file comparison method. I coded to compare the files in two ways. One is to generate a digital hash (SHA-512) of the files and check against other folders and files. If the file size is big, this method would slow down your computer, and it's better to go with the second method. The first method is good for text files and files that are too important. If you want to choose the first method, click on "Yes". The second method is basically checking the file name and size only. If two different images have the same file name and file size but are in two different folders, these two files will be identified as duplicate files with the second method. But the second method is much faster for the utility to scan the folders and identify duplicates. So, if you have less important big movie files, choose the second method by selecting "No".

5. Now the utility will scan for duplicate files and folders and show you the percentage completed.

6. Once the scanning process is completed, if there are duplicate files, they will be listed in a separate window for you to click on the checkbox and delete.

7. Once you click on a checkbox, you will be prompted to confirm, and once you choose "Yes", the file or folder will be permanently deleted from your HD.

I've tested this utility a few times on my machine by creating dummy folders and files and couldn't find any issues yet. I deleted many files on my local external disk using this utility. Please check it out and let me know if there are any issues.

I do not accept any responsibility and will not be liable for any loss or damage suffered by you whatsoever arising out of the usage of this utility.

Road Accident Lessons: What I Learned the Hard Way

I've been driving for over a decade, having obtained my driver's license in 2006. Now in my early thirties, I've always prided myself on being a cautious driver.

On May 1st, 2017 (Labour Day), I found myself with a rare day off. After sleeping in, I called my girlfriend only to discover she was unwell, suffering from severe gastric pain and a headache. I drove to her apartment to check on her, then headed to Arpico Pharmacy in Kiribathgoda to purchase medication.

After buying the medicine and some groceries, I started my journey back. Unable to make a U-turn at the junction next to Arpico, I took a right turn onto 'Aluth Para' towards 'Hunupitiya'. I safely parked on the roadside, crossed the street to buy a water bottle, then returned to my car. After carefully checking all directions, I began a slow U-turn. The holiday meant the roads were unusually quiet, and I was driving cautiously, savoring the peaceful day.

As I was about two-thirds through my U-turn, I heard a sudden horn blast. I immediately hit the brakes, but within a split second, I saw a young man on a motorbike collide with my car. Unable to control his speed, he fell, sliding past my vehicle. I was in shock.

The motorcyclist, despite his injuries, jumped up and approached my car, shouting angrily in Sinhalese. The shopkeeper who had sold me the water came out to see what had happened. I exited my car to assess the situation. The young man seemed more concerned about the motorcycle (which belonged to his elder brother) than his own injuries. His hands and legs were scratched, and he was visibly shaking from pain and shock. I offered to drive him to the hospital, but he refused, fixated on how he would explain the damaged bike to his brother.

Soon, a crowd gathered. An older man called for an ambulance, initially against the rider's wishes. The shop owner mentioned seeing the motorcyclist speeding while I was turning slowly. When I asked if she would testify to this, she initially agreed but later declined when the police arrived.

The elderly man (about 50-60 years old) called the rider's brother, explaining the situation calmly. My limited Sinhalese made communication difficult, and I sensed hostility from the crowd, possibly because I was uninjured while the rider was hurt. I remained quiet, unable to explain myself adequately.

When the ambulance arrived, the rider initially refused to go, still upset about the bike. His brother soon arrived, checked on the bike's status, and took my phone number. The injured rider finally agreed to go to the hospital.

I waited for my insurance representative while the police took my license and instructed me to come to the station. The shop owner commented on young riders' recklessness but was unwilling to officially testify. I called my girlfriend to explain what had happened.

At the police station, an officer asked me to drive him to the hospital to check on the motorcyclist's condition. We discovered he wasn't at Kiribathgoda hospital. Back at the station, I waited until about 8:30 PM, giving my statement in broken Sinhalese. The possibility of spending the night in custody loomed until a higher-ranking officer allowed me to leave my car and keys, instructing me to return in the morning.

[Image placeholder: Generic police station]

The next day, I returned to the station at 8:30 AM. The rider's brother arrived, and negotiations began. When asked for compensation, I offered 3,000 rupees, hoping to settle around 5,000. The brother was offended, demanding 30,000 rupees. Frustrated by the assumption of my guilt, I finally found the courage to explain my side to the police officer in my limited Sinhalese.

By noon, I was told to go to Mahara court. Angry at what I perceived as an unfair system, I decided to fight the case, naively believing I could present my side of the story effectively.

After 11 months of legal battles, I gained a better understanding of the system's inner workings. I was never allowed to speak freely in court, faced challenges with lawyers, and encountered numerous frustrating situations that interfered with my daily life.

This post is meant to share my experience and offer advice for those who might face similar situations. I've intentionally omitted many infuriating details from the past 11 months.

Key learnings from this accident:

1. In accidents, provable facts matter more than what actually happened.
2. Overcome shock quickly and gather evidence immediately.
3. Control anger and communicate clearly, especially with the police.
4. Avoid confrontation and arguments.
5. Try to reach a calm settlement before police arrive, if possible.
6. Collect evidence and take photos of the accident scene.
7. Gather all relevant information (driver details, insurance info, vehicle details, etc.).
8. Document accident specifics (time, location, road conditions, etc.).
9. Don't communicate with the other party's insurance company without legal counsel.
10. Don't sign any documents without lawyer review.
11. Install a dash cam in your car.
12. Prepare a comprehensive statement for the police.
13. Have support from friends or family at the police station and court.
14. Seek help finding a good lawyer from the start.
15. Try to maintain a good relationship with the police.
16. Choose a reputable insurance company.
17. Stay hydrated and energetic despite stress.
18. Prepare for the worst while maintaining a positive attitude.
19. Drive defensively, anticipating others' mistakes.
20. Keep emergency contacts and important documents together.

Remember, if you're in an accident, try not to panic. Time changes everything, and this too shall pass.

Thank you for reading, and stay safe on the roads.