![]() Files are added to the current chunk up until the uncompressed size reaches a certain threshold (512 KB at the moment) and then the chunk is compressed and a new chunk is started. ![]() It’s stored in one file that consists of chunks each chunk contains a list of file paths and file data, both compressed using LZ4. To get around these issues, qgrep maintains a compressed representation of the entire searchable codebase. As you can see by looking at ripgrep results, the impact is very significant even on SSDs on HDDs we’re talking about waiting for a minute for the search to complete. Even when filesystem hierarchy was in the cache, retrieving the hierarchy wasn’t very efficient either due to a large number of kernel operations required. In theory, the filesystem in-memory cache is supposed to make subsequent searches fast on FIFA codebase, however, both file hierarchy and file contents was never fully in the filesystem cache, especially as you worked on the code (for example, switching to the browser to google something would likely evict large portions of the search data set). A recursive grep tool must start by recursively traversing the target folder as new files are encountered, it needs to read their contents and perform regular or literal search on the results, and print the matches if any. The single biggest source of performance issues at the time was disk I/O and I/O-related system calls. But how is it possible to make a code search tool that’s so much faster than ripgrep? Why, by cheating, of course. It’s possible that if in 2012 I used SSDs and ripgrep was available, I wouldn’t have gone through the trouble of making a new tool - but the pain threshold was crossed so I did and here we are. Search complete, found 1 matches in 0.03 secĬ:\work\unrealengine>rg -stats "vkCmdDrawInde.*KHR = 0"Įngine\Source\ThirdParty\Vulkan\Include\vulkan\vulkan.hppĥ3924: PFN_vkCmdDrawIndexedIndirectCountKHR vkCmdDrawIndexedIndirectCountKHR = 0 Īnd here’s the table of timings in seconds here we also run qgrep with a b option that will be explained later (it stands for bruteforce): tool Here’s the hot run:Ĭ:\work\qgrep>build\Release_圆4\qgrep search ue4 S "vkCmdDrawInde.*KHR = 0"Ĭ:/work/unrealengine/Engine/Source/ThirdParty/Vulkan/Include/vulkan/vulkan.hpp:44478: PFN_vkCmdDrawIndexedIndirectCountKHR vkCmdDrawIndexedIndirectCountKHR = 0 The timings are done using my desktop system with i7 8700K (6 cores 12 threads) and Samsung 970 EVO SSD (which is an M.2 NVMe drive) on Windows 10, using latest ripgrep/qgrep 圆4 builds. We’ll run qgrep and ripgrep twice: first time is after a reboot so the filesystem cache is cold, and second time is immediately after that. Still, qgrep is much faster.Īs an example, let’s try to search for a simple query, vkCmdDrawInde.*KHR = 0, in UE4 codebase which is similar in size today to FIFA codebase from 2012. Today the golden standard of performance is set by ripgrep, which runs much faster than the alternatives, and the norm is to use an SSD. Remember, qgrep was written in 2012, so the competition at the time was grep, ag, Visual Studio Ctrl Shift F, and the hardware target was a multi-core CPU with an HDD. This article will go over the design and performance optimizations that make qgrep really fast. ![]() Thus I decided to make a new tool, qgrep 3. At the time, the only existing tool that gave near-instantaneous results was Google Code Search - unfortunately, the performance on case-insensitive queries or some types of regular expressions at the time wasn’t very good 2. I wanted a tool that was much faster than existing alternatives 1 and was able to perform both literal and regular expression searches at similar speed. This was frustrating and as such I decided to solve this problem. Using Visual Studio Ctrl Shift F search on a HDD on a codebase this size means that every search takes minutes. Given an unknown codebase, you need a way to quickly get around it - since you don’t know the code, you resort to search-based navigation, aka grep. In 2011-2012 I worked on FIFA Street, followed by FIFA EURO 2012 DLC and finally FIFA 13 - all of these games were based on the same codebase, and this codebase was HUGE.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |