Help me clean up data sources

Hello, I'm an experienced Linux user and Python developer. I've worked with elasticsearch, Postgres, MongoDB. Depending on the data you have and the current bottleneck there might be a few solutions: - for example if your IO subsystem is fast enough and you have multiple cores you can split the work into multiple subprocesses - depending on the data and typical queries you can insert the data into a database and index it; elasticsearch might be a good option here. To help identify the bottleneck tools like htop, dstat can prove very useful. Can you give me more details about the text files? Best wishes, iticus

Hi there, I'm a professional Big Data software developer. I'm having expertise in developing software using Hadoop, Spark, HBase, Scala, Cassandra etc. See some of the works in my portfolio. I will develop a map-reduce application which will accomplish the job quickly. Ping me for discussion.

Hi, I'm a french software engineer, and I new to this website. I just create my compagny, and I develop software for compagnies. I worked for 10 years as a software developer. For my first project on this site, I would be very happy that this is yours. We can transform your file into XML or use a database. I am available from now, and for the time you want. If you have any questions do not hesitate to contact me. Best Regards, ilatech team

Hi, I have read the project description and understood your requirements. In order to make your project a success, you need someone who has ample experience in Data Warehousing (organizing large amount of data in a format that is easy to retrieve required information) and a keen eye for detail, both of which I have. I will work efficiently and can start the work at the earliest with a guarantee of work to your satisfaction. I can do a sample right now and we can discuss the budget if you are satisfied with my work. Hopefully I can offer you my service and contribute to your success. Regards, Sandeep.

Hi, I am Amit. For the problem definition you have provided. I think it will be better if we can use Hadoop framework for the solution. We can use either MapReduce or Spark for processing. Also, Please tell me the processing that needs to be done once we get a match. Regards, Amit

I think you need the power of parallel processing to achieve what you want. It all depends on the type of data in the files and the average size of each file. We may opt for an installation of Hadoop or any of its free distributions. We will need to set it up on a cluster. You do not need a lot of physical machines for that. We may virtualise a single machine with multiple servers each of which will contain a node of Hadoop. If the data is semi structured, you may use Hive to query it. If you need custom searching, a client may well be written over Hive. Else you may also use PIG Latin. An alternative approach will be to install a cluster of NOSQL database like Cassandra to hold your data. Again, this depends on the structure of your data. Whatever the option maybe, this project needs a thorough analysis phase to arrive at the right choice for you. Please be aware that the time and cost may vary based on the outcome of the analysis.