Skip to content
  1. Feb 29, 2016
    • Carlos Garnacho's avatar
      Update to FTS5 · 93c149a5
      Carlos Garnacho authored
      Our old stale copy of the FTS3/4 module is now deleted, replaced by
      a shinier FTS5 embedded module. If at configure time we detect that
      SQLite doesn't offer the FTS5 module, we will load our own, just as
      we used to do with FTS4.
      
      FTS5 brings a few differences in the ways it's meant to be extended,
      the tokenizer has been updated to cope with the differences. Also,
      FTS5 offers no offsets() builtin function, nor matchinfo() which we
      used to implement ranking. It offers though ways to implement
      additional functions, and builtin rank support which can be tweaked
      to achieve the same functional results than we did.
      
      Other than that, the ways to interact with the FTS virtual table
      are roughly similar to those in FTS4, insertions and deletions have
      been updated to do things the FTS5 way.
      
      Since it's not worth to bump the database format (data is reproducted
      from the journal, so we drop some embedded data such as
      nie:plainTextContent), the nco:hobby property has been modified to
      no longer be fulltext indexed, AFAIK there's no users ever setting/
      accessing that, and the FTS properties change will trigger the
      regeneration of the FTS view and virtual tables, resulting in a
      seamless update to FTS5.
      
      However, we don't leave completely unscathed from the fts3_tokenizer()
      change. Since the older FTS3/4 tokenizer is not registered, we can't
      just drop the older FTS table. So it is left dangling and never
      accessed again, in favor of the newer fts5 table. This is obviously
      not a problem when creating the database from scratch.
      
      In the way, a few bugs were found. per-property weights in ranking
      were being given in a scrambled way (although stable across database
      generations). And deletion of FTS properties (or entire rows) could
      result in the tokens not being fully removed from the FTS table,
      resulting in confused searches. These are now fixed.
      
      Impact to users of tracker should be none. All the FTS Sparql-to-SQL
      translation has been updated to just use FTS5 syntax and tables.
      93c149a5
    • Carlos Garnacho's avatar
      libtracker-data: Handle tracker:fulltextIndexed changes · b324a0fd
      Carlos Garnacho authored
      We no longer have a reason to deem this change incompatible. Since
      no actual text is stored in the FTS tables, and it merely updates
      its tokenization info from the original class tables, we can just
      drop this info and reconstruct at will. So there's no longer need
      to consider tracker:fulltextIndexed changes as incompatible.
      b324a0fd
  2. Feb 26, 2016
  3. Feb 25, 2016
  4. Feb 24, 2016
    • Carlos Garnacho's avatar
      libtracker-miner: Do not break ordering with bulk operations · c0310dbb
      Carlos Garnacho authored
      The way bulk operations break the ordering of the sparql updates
      may bring certain glitches. eg. if a delete (bulk) operation
      happens on a folder while the sparql buffer already contains
      updates corresponding to files within that folder, the way we
      order operations results in the delete happening first, and
      the inserts/updates happening afterwards. This may leave the
      database in inconsistent states.
      
      So make bulk operations a 1 file thing, this could maybe be
      smarter and try to compress consecutive similar operations,
      but there's not much added value in it. Plus, the bulk operations
      we issue in the miner would never match that criteria.
      
      Because we no longer break the ordering with bulk operations,
      things like the error map are no longer necessary, so we can
      remove some icky code here.
      
      BulkOperationMerge and its functions could be further simplified,
      that's left for a later cleanup.
      c0310dbb
    • Carlos Garnacho's avatar
      libtracker-miner: Cancel ops in current buffers if a delete affects the file · 0eb42084
      Carlos Garnacho authored
      If a folder being deleted affects operations currently in the currently
      issued tasks (eg. those we emitted ::process-file on) and writeback buffers,
      those operations would still attempt to proceed, with different degrees
      of success.
      0eb42084
    • Carlos Garnacho's avatar
      libtracker-miner: Stop crawling if a delete op affects the current directory · 9a8c53ae
      Carlos Garnacho authored
      Otherwise the crawler will still attempt to go through the processed
      folders, adding unnecesary processing and potentially leaving inconsistent
      state in the TrackerFileSystem if a similar file layout appeared in the
      future.
      9a8c53ae
    • Carlos Garnacho's avatar
      libtracker-miner: Fix TrackerFileNotifier cancellable maintenance · 01958c04
      Carlos Garnacho authored
      Resetting and reusing is not deemed safe. It is better to create new
      cancellables and let the old ones be last unref'ed after async callbacks
      finish.
      01958c04
  5. Feb 23, 2016
  6. Feb 15, 2016
  7. Feb 14, 2016
  8. Feb 06, 2016