Removing duplicate files and folders - Cleaning up my hard disks

Ever since I started using a computer about 20 years ago, I carefully collect all the files thinking I would need in the future and quite frankly, I absolutely needed none of the file I created before 10 years. Those days I was hoping that in the Job interviewers might ask me to show the works I have done, but they never did. Having had the chance to see the HR processes from inside the organization in few of my previous jobs, I could tell, one should be lucky if an employer had the mindset to actually check whether an applicant has done something in the past and what they have done. All these new fancy HR processes make people too busy having to complete too many interviews for a position open, for reporting sack, they do not have the time to actually go and check an applicant's past projects and the quality of the work.

My habit to collect and save all the files that I make for the projects has gotten to a point I need multiple high-capacity hard disks and cloud space. I save my project files, important tutorials, camera raw files, picture/video edits, templates, songs I like, movies I like and many other files including automated back-up every three months. My digital hoarding disorder has gotten out of proportion to a level, I have a back-up of back-ups on Google clouds and important documents on one drive.

So, now I have about 1 TB cloud-space, 8 TB local external storage options on multiple devices and a bunch of SD cards and pen drives could be up to a total of 500 gigs. Whoever would read this blog post five years or ten years from now would smirk at these numbers as we now laugh at the 20 gig HD on our old P4 computer (Millennium Super computers: 😛) now, but for the present-day specs, I think as an individual PC user trying to keep this much of data is crazy. At the place I worked for in 2005, the whole organization had about 2 TB space to manage the data.

Last few months, most of my weekends got wasted in my efforts to organize these files and eliminate duplicate files and folders. Within a few hours of starting this task I figured that it's not humanly possible to go through these thousands of folders and files one by one and delete the duplicate files. I was not ready to trust the exe HD clean-up utilities on the internet either. So, with few googlings, I wrote up a bat code that could loop through the file list and identify the duplicate files and print the details on the cmd screen. Again, it was difficult to go one by and delete, also dangerous to let the program automatically delete all at once. 

After few more attempts to modify the bat file code to selectively delete the files and folders by prompting the user, I found the code to be messy, complex and not very user-friendly for the user to go one by one again. By having the same bat logic as the core, I wrote a Java program with few classes achieving much better usability. Struggled a lot to fix the GUI freeze when thread is busy problem. When the java.io is busy in the current thread reading the files or in separate thread, the GUI becomes non-responsive and we cannot see the progress of the process. After drilling Stack Overflow with all what I got, I found out that the SwingWorker is the answer to solve the problem. From there, it was easy peasy, within a few hours the "JJCleanFF" :) utility is ready and was able to delete a lot of duplicate folders and files.

For now, I share here the jar file and instructions to use the utility with you, after cleaning up the code a little bit and adding documentation, will post on GitHub and share the link in this post for source code.  So,

1. Download the executable jar file by clicking below icon or visiting https://www.jeyaramj.com/downloads/JJCleanFF.jar

2. Open the downloaded jar file.

3. In the main window, click on "Select a folder" to select the parent folder in which the folders and files be checked for duplicates

4. Once the folder is selected, there will be a message prompt for you to choose the file comparison method. I coded to compare the files in two ways. One is to generate a digital hash(SHA-512) of the files and check against other folders and files. If the file size is big this method would slow down your computer and better go with second method. First method is good for text files and files that are too important. If you want to choose the first method, click on "Yes". Second method is basically checking the file name and size only. If two different images with same file name and file size, but in two different folders, these two files will be identified as duplicate file with the second method. But second method is much faster for the utility to scan the folders and identify duplicates. So, if you have less important big movie files, choose second method by selecting "No".

5. Now the utility will scan for duplicate files and folders and show you the percentage completed.
  

6. Once the scanning process is completed, if there are duplicate files will be listed in a separate window for you to click on the check box and delete.

7. Once you click on a check box, you will be prompted to confirm and once you choose "Yes", the file or folder will be permanently deleted from your HD.

I tested few times in my machine by creating dummy folders and files and couldn't find any issues yet. I deleted many files on my local external disk using this utility. Please check and let me know, if there are any issues.

* - I do not accept any responsibility and will not be liable for any loss or damage suffered by you whatsoever arising out of the usage of this utility.

Met an accident last year. My experience might be useful to you.

I got my driver's license in 2006 and now I’m driving about 10+ years. I’m in my early thirties and always take extra caution while driving.  I consider myself a safe driver. 

As I was working hard on the weekdays, preparing for my wedding and other things were going on in my life on the weekends, I rarely find time for myself. On May 1st 2017(Labour Day), which is a holiday, I woke up late and called my girlfriend and found that she was sick and suffering from severe gastric and headache. I went to meet her in the car to her apartment and after speaking to her, I went to buy the gastric medicines in the Arpico Pharmacy in Kiribathgoda.

After I bought the medicine and other groceries, I went towards Colombo side to come back to Kiribathgoda Junction, but as we cannot take ‘U’ turn in junction next to Arpico, I took a right turn and went in the ‘Aluth Para’ to ‘Hunupitiya’ and stopped safely on the side of the road without blocking any vehicles. Stopped the vehicle and went to the other side of the road and bought a water bottle. Came back to my car and started the car and took the ‘U’ turn after looking all the sides. Since it was a holiday, there were no vehicles on the road and I took the car very slowly. Since I was so relaxed and wanted to have a peaceful holiday, I was driving very slow throughout. 

As I was completing more than two third of my ‘U’ turn, I heard a horn and I suddenly braked. Within a split second, I saw a young man hit a motorbike on my car and unable to control the speed, went and fell passing my car. I was so shocked. As I was in the car, the young man who fell(who was riding the motorbike) got up and come in front of my car and shouted at me using bad words in Sinhalese. The small shop lady from whom I bought the water, came out and I also got out and observed the situation. He was so upset and worried about the bike more than his wounds and telling me that it is not his bike and his elder brother’s bike and how could he tell this to his elder brother and was shouting at me standing. His hands and legs had some scratches and were shaking out of pain. I offered to take him to the hospital in my car. He refused and kept repeating that how could he repair his bike. The shop lady shouted saying "go to the hospital and treat the wounds, don’t know the inside wound". 

After a few minutes, many other people gathered and there was an old man who called the ambulance to take him to the hospital. He initially refused and then agrees as the old man insisted saying that he will talk to the brother. The shop lady told me that he came very fast and she saw me turning slowly. I asked her as to whether she would be ok to testify for me. She said she can’t. I asked her, "at least would you testify for me for turning very slowly?". She said yes first and then, when the police came, she didn’t want to. The old man asked my phone number and I gave my number to him. The old man (about 50 to 60 years old) called his brother and told that there had been an accident and that we were sending him to hospital, and there's nothing to worry. 

I had real problem communicating to them in Sinhalese. The old man noticed and asked me in English ‘Did you call your insurance?’ I said 'yes'. Then there was a policeman near to the area came and assessed the situation. The police officer asked me some questions. I answered in Sinhalese to my ability. I was able to feel that everyone is angry at me even without looking at the accident. I thought that this must have been due to the fact that I had no injury and the motorbike rider got injured. Due to my limited language skill, I couldn’t explain myself to anyone. I kept quiet. I did not tell anything to the rider as he got wounded and I did not want to tell anything in my broken Sinhalese. Then the ambulance came and they asked them to go in the ambulance and he was refusing to go as he was upset about the fact that this happened and didn’t know how to answer his brother. Others were insisting him to go in the ambulance.  Then in about 5-10 mins his brother came and talked to him and asked about the bike. The others said that it is in the service station. He came to me and asked my phone number. I gave him my phone number. His brother talked to the police and went while the motorbike rider who got wounded went in the ambulance. By looking at the wounds, which I noted while he was standing and shouting at me immediately after the accident, I thought he will be fine after treating the wounds(scratches). 

Everybody went and I was waiting in the same spot for the insurance to come. The Police took my license and told me to come to the Police Station and left. Only the shop lady was around and she was saying that these young motorbike riders are too fast. Again, I asked her whether she would be willing to tell the police what she saw, but she was not ready to do that. I called my girlfriend and told what happened. She was already sick and now got really upset, but she said not to worry. At the police station, a police man asked me to come to the hospital with him in my car to see the motorcyclist condition. I went with the police to the hospital. The police came back and told me that he is not in the Kiribathgoda hospital.

I came back to the police station and waited there till about 8.30 PM. The accident happened around 5 PM. The Police Officer took my statement. I was worried as to whether he understood my Sinhalese and I couldn’t read what he’s writing as it was written in Sinhalese. He told me that I will have to stay in the Police Station that night as the police is unable to contact the Motorcycle rider or his brother to find out what happened to him. I was shocked and got very scared. I called my girlfriend immediately and she took a three-wheeler and come to police station and talked to the police officer. After some time, there was another police officer in a higher ranking who called us and told to leave the car and the car key and to go home and come back tomorrow morning. It was a big relief. I went to drop her in her apartment and came home with absolute sadness and tired.

Next day morning I went to the police station at about 8.30 AM. The brother of the motorcycle rider came and spoke to the police. I had spoken to the policeman who inspected the accident and found out that I might have to pay about four or five thousand to the rider to make a deal. The brother came and told me that this is an accident and he knows that these things happen and asking me whether I can pay him. I asked him how much he wants. He told me to tell a fair amount. After a few times asking him to tell the amount, I said to him, I’ll give him 3,000 expecting to settle around 5,000. He got offended and told me that he wants 30,000 rupees. I was shocked. In my mind, I still couldn't see why all of them were treating me as someone who's guilty. I was so upset and tired. I understood that the fact that I’m not talking much to the police was making the situation worse. Though there were many people, I found the courage to talk to the policeman in my broken Sinhalese and told that it was not my fault as he was very fast and that I didn’t know how I could have avoided what happened. The road was not a straight one so I couldn’t see him in the beginning. He came very fast and unable to control his bike tried to overtake my car in a dangerous way and his motorbike guard bar hit my car and got out of balance and fell and got the scratches. The police officer told me that they do not know that at what speed the motorbike came. I was so upset by the police officer's comment and felt hopeless and came back and sit in the chair. I called my reporting manager at work and requested for a day off. After passing 12, the police officer told me to come with them and told my girlfriend to take some money and come to the Mahara court. 

I was angry towards the unfair legal system, irresponsible responses from the police and the circumstances in my personal life at the time. I wanted to fight in the court believing that I could tell my side of the story and ask how I could have avoided the accident or how it's fair to frame me guilty without evidence.

I was a fool and naive to think that. After about 11 months of fighting in the court, I've come to a better understanding of the inner workings of the system, practices, lawyers' part, police's part and how all that were just feeding my ego nothing else. 

I was never allowed to speak freely in the court; most lawyers were too busy to hear me out; a lawyer didn't come on time to the court and got me an arrest warrant but still blamed me and charged me for his time; couldn't find a lawyer who would tell to the court that the other lawyer was late and that was why I was not present when my case was called; and, there were a million things that made me just angry and were interfering with my day to day life. I was just wasting my time and money while costing my family & friend's happiness just to prove my idiocy. This was all just to console my anger that I was feeling towards more than just one thing.

This post is just to share my experience and help you out should you come to face this kind of situation. I purposely left out many other things that made my blood boil and had to go through last 11 months.

If I were to list out some of my leanings through this accident.

1. When it comes to accidents, it's not about what happened, it's about what happened that you can prove.
2. When you are met with an accident, try to quickly get over the shock that you might feel at that moment and start acting quickly to understand the situation and collect evidence.
3. If you were like me, anger will silence you at the most important time you need to speak. So, try controlling your anger and speak loud and clear with everyone at the spot, especially with the police.
4. Confronting and arguing never helps, I think. 
5. Coming to a settlement clammily discussing, before the police arrives, would be the best idea, but when one is injured and out of control, it might be tough.
6. Collect evidence and take photos of the accident. (Name of Driver, Address of Driver, Phone number or other contact information)
7. Collect information about the Other Driver's Insurance Company (insurance company name, insurance company policy number)
8. Collect information about the Other Car (Vehicle Description, Make, Model and Year - Consider taking photos for your own record if it is possible. Vehicle registration information, License plate number)
9. Other information to collect
1. Date and Time of the Accident
2. Address of the accident, or approximate address
3. Road you are on and the nearest cross street
4. The direction you were traveling in
5. The direction the other car was traveling in  
6. Take photos from a few angles or sketch a diagram of the crash scene
7. What happened 
8. Any notes regarding the driving conditions (try to figure out whether the other person is under the influence of alcohol), the weather, visibility
9. Any witnesses' names and contact information
10. The name, badge number, and contact information of any police officer who comes to the accident spot
10. Don’t talk to other driver’s insurance company – Do not give a recorded statement to the other driver’s insurance company without counsel being advised and present.
11. Don’t sign anything – After the accident, you may be asked to sign releases from the other driver’s insurance company. Do not sign any documents before your lawyer has had a chance to review them.  (Thanks reynoldsinjurylaw.com/8-things-to-consider-if-you-are-in-a-car-accident/)
12. Have a dash cam fixed to your car.
13. If it has to go to the court, come up with everything that you need to tell to the police that would be included in the police statement.
14. Always have your friends or family members by your side to help you at the police station or at the court.
15. Seek help from your friends or from me to find you a good lawyer from the start. (I made a big mistake on this.)
16. Try your best to keep the police on your side.
17. Do your research and get the insurance from a friendly insurance company.
18. Disappointments and circumstances might make you feel tired and lethargic, don't be. Take enough fluids keep yourself energetic at all times.
19. Always prepare for the worst and act as if everything is in your favour. 
20.  Drive very safe not expecting others to do the right thing on the road. But, don't be scared or agitated when driving.
21. Have some emergency contacts (Family member or friend to call, lawyer, insurance contact, service station number for quick quotation etc..) with your tax, insurance and driver's license. 

If you got into any kind of accidents, don't worry too much immediately and panic. That would make things worse and remember Time Changes Everything.

Thank you for checking this post and wish you well.