Displaying huge text files

-
Displaying huge text files

Sometimes programmers have to deal with larger than usual text files. Examples are log files or XML dumps of databases. For my tests, I used a dump of the English Wikipedia in XML format. This file is 48 gigabytes in size and as I found out only today contains 789,577,286 lines of text.
http://blog.dummzeuch.de/2016/05/29/displaying-huge-text-files/

Comments

  1. A. BouchezMay 29, 2016 at 12:08 PM

    My good old TotalCommander has no problem opening and displaying a 50 GB (text) file, instantly. Just avoid the Windows notepad. ;)

    ReplyDelete
  2. Joseph CarneyMay 29, 2016 at 12:51 PM

    Not being sarcastic, but I would have thought a lot more than 48 gigs. Surprised.

    ReplyDelete
  3. Bill MeyerMay 29, 2016 at 3:24 PM

    Back in the days of CP/M, my favorite text editor was able to open files of over a MB in size. For the kids in the audience, it did this with only 62KB of ram in the workspace. It accomplished this by implementing virtual memory very cleverly, and while I was not privy to the details, I used it heavily on code, and found it always responsive and reliable.

    In these days of cheap RAM, most people simply assume they can load the entire text into RAM. Simple, and dumb.

    ReplyDelete
  4. Thomas Mueller (dummzeuch)May 29, 2016 at 10:00 PM

    Bill Meyer it's not dumb, it's using the resources available for best effect. Let's face it, most text files are only a few kbytes in size, and most people will never open anything else. They need a tool that has other features than displaying huge files.

    ReplyDelete
  5. Daniela OsterhagenJune 1, 2016 at 12:40 PM

    Joseph Carney Text, even in XML Form, is just not that large. Most of Wikipedia's data volume probably is pictures.

    ReplyDelete

Post a Comment