What is ExcelAnalyzer
developXLSX files for hidden data. It is intended to work with files uploaded and analyzed as ActiveStorage::Blob objects. The analyzer identifies hidden data and appends the findings to the blob's metadata.repository·develop·Indexed 19 days ago
https://github.com/mysociety/alaveteliAn open-source, internationalised platform for facilitating Freedom of Information (FOI) requests globally to promote transparency between citizens and governments. The project includes the Alaveteli core application, the alaveteli_features gem for feature management via Flipper, and the ExcelAnalyzer gem for inspecting XLSX files for hidden data.
XLSX files for hidden data. It is intended to work with files uploaded and analyzed as ActiveStorage::Blob objects. The analyzer identifies hidden data and appends the findings to the blob's metadata.Alaveteli uses a search_documents table to store searchable data, which decouples the search index from the primary model tables. This allows for efficient searching using PostgreSQL's ts_vector capabilities.
There are two main ways to handle indexing:
object.reindex: Upserts content for a specific instance into the search_documents table. This should be called whenever a model's searchable content is modified.Model.reindex_all: Overwrites all existing entries for a specific model in batches. This is useful for initial backfilling or full refreshes and can be run without service interruption.Note: Reindexing large datasets (e.g., FoiAttachment) can take significant time (minutes to days).
# Reindex a single instance
PublicBody.find(123).reindex
# Reindex all instances of a model
PublicBody.reindex_allThe Search module acts as a decoupled facade that allows the application to interact with search engines without being tied to a specific implementation. It uses a pluggable backend architecture:
Search): The entry point for controllers, models, and mailers (e.g., Search.search).Search::Backend): Defines the interface that all concrete implementations must follow.Search::Adapter): Concrete implementations (like Adapters::Xapian or Adapters::PostgreSQL) that map generic search requests to backend-specific logic.FullText, Typeahead, and SimilarRequests are handled by adapters and return a unified Search::Results object.Controllers / Models / Mailers
|
v
Search module (app/search/search.rb) -- public facade
|
v
Search::Backend (app/search/backend.rb) -- abstract interface
|
v
Adapters::Xapian (app/search/adapters/) -- concrete implementation
|
v
Search::Adapter (app/search/adapter.rb) -- base for search types
|
+---+---+------------------+
| | |
FullText Typeahead SimilarRequests
| | |
v v v
Search::Results (app/search/results.rb) -- unified result objectTo make a model searchable, add a searchable definition to it. This definition specifies which fields (or Ruby methods) should be included in the search index and assigns them weights (A being the highest) for ranking results.
.name) to call a Ruby method.home_page) to access the PostgreSQL column, which is faster but less flexible.You can define both a public index and an admin_index for content only visible to administrators.
searchable index: {
".name": "A",
"home_page": "B",
},
admin_index: {
# same logic as above
}To set up the development environment for the excel_analyzer gem, follow these steps:
bin/setup
bin/consoleStandard view templates must be renamed from the .rhtml extension to .html.erb. You can automate this in your theme's root directory using the following command:
Note: Mailer templates are an exception; they should be renamed to .text.erb because Alaveteli uses text-only emails.
for r in $(find lib/views -name '*.rhtml'); do echo git mv $r ${r%.rhtml}.html.erb; doneTo add alaveteli_features to your Alaveteli application, add the gem to your Gemfile and run bundle, or install it directly via the gem command. After installation, run the generator to set up necessary migrations and an example initializer file.
# Add to Gemfile
gem 'alaveteli_features'
# Then run in terminal
$ bundle
$ rails g alaveteli_features:installAlaveteli is tested against ruby-3.4.
If you are using a Ruby version manager like RVM or .rbenv, you can automatically switch to the recommended development version (currently 3.4.7) by creating a .ruby-version symlink pointing to .ruby-version.example within the project directory.
ln -s .ruby-version.example .ruby-versionTo implement a new search backend, follow these steps:
Search::BackendCreate an adapter class (e.g., app/search/adapters/my_backend.rb) that implements the required interface:
search(query, models:, sort_by: nil, sort_ascending: true, collapse_by: nil): Must return an object responding to .results(page:, per_page:) which returns a Search::Results object.typeahead(query, model:, exclude_tags: []): Must follow the same contract as search.similar(record): Must return an object responding to .results or .first.Each method on your adapter should return an object that implements the results logic. You can use Search::Adapter as a base class to help create Search::Results objects using create_search_results.
Assign your new adapter to Search.backend in an initializer, or set the SEARCH_BACKEND value in your configuration (e.g., config/general.yml).
# In an initializer
Search.backend = Search::Adapters::MyBackend::Adapter.newIf your backend requires specific indexing setup (like PostgreSQL triggers), place the configuration in your adapter's namespace (e.g., Search::Adapters::MyBackend::Indexing) and call it from an initializer.
module Search
module Adapters
module PostgreSQL
class Adapter < Search::Backend
def search(query, models:, sort_by: nil, sort_ascending: true,
collapse_by: nil)
# Return an object that responds to .results(page:, per_page:)
# and returns a Search::Results
end
def typeahead(query, model:, exclude_tags: [])
# Same contract as search
end
def similar(record)
# Return an object responding to .results / .first for records
# similar to the given one
end
end
end
end
endIf your theme patches mailer paths in lib/patch_mailer_paths.rb, update the method used to add view paths:
ActionMailer::Base.view_paths.unshift ... with ActionMailer::Base.prepend_view_path ...ActionMailer::Base.view_paths << ... with ActionMailer::Base.append_view_path ...# Old
ActionMailer::Base.view_paths.unshift File.join(File.dirname(__FILE__), "views")
# New
ActionMailer::Base.prepend_view_path File.join(File.dirname(__FILE__), "views")/rails/mailers endpoint in your local development environment.