The Great Rename

Rename the project to quran-json

.gitignore

@@ -1,2 +1,3 @@
+share/quran-json/TheQuran/
 *.db
 

README.md

@@ -1,62 +1,49 @@
 ## About
 
-This repository contains the contents of the holy book, The Quran - in its original
+quran-json is a dual purpose project where on the one hand it provides a utility
-Arabic. Translations in English, Farsi, and Portuguese are also included. The contents
+for downloading the content of The Quran in multiple languages, and on the other hand
-are available in the JSON, and SQL formats.
+it provides a copy of the content that can be downloaded. In both scenarios, the content
+is provided in the JSON format.
 
-**Contents**
+## <a id='share/quran-json'>share/quran-json/TheQuran/</a>
 
-1. [src/json/](#srcjson-directory)
+* [share/quran-json/TheQuran/ar/](share/quran-json/TheQuran/ar/) contains The Quran in its original Arabic.
-2. [src/sql/](#srcsql-directory)
+* [share/quran-json/TheQuran/en/](share/quran-json/TheQuran/en/) contains an English translation of The Quran.
-3. [bin/](#bin-directory)
+* [share/quran-json/TheQuran/fa/](share/quran-json/TheQuran/fa/) contains a Farsi translation of The Quran.
+* [share/quran-json/TheQuran/pt/](share/quran-json/TheQuran/pt/) contains a Portuguese translation of The Quran.
 
-## <a id='srcjson-directory'>src/json/</a>
+### JSON layout
 
-* [src/json/ar/](src/json/ar/) contains The Quran in its original Arabic.
+The
-* [src/json/en/](src/json/en/) contains an English translation of The Quran.
+[share/quran-json/TheQuran](share/quran-json/TheQuran/)
-* [src/json/fa/](src/json/fa/) contains a Farsi translation of The Quran.
+directory contains multiple sub-directories, where each sub-directory represents
-* [src/json/pt/](src/json/pt/) contains a Portuguese translation of The Quran.
+a locale (eg `en` for English, `ar` for Arabic, and  so on). Within each sub-directory,
+there is a JSON file for each surah (also known as a chapter).
 
-#### JSON schema
+The structure of each JSON file can be described as an array where the first element is
+an object that contains information about a surah, and the rest of the array contains
+the content of the surah. The content is composed of two-element arrays - where the first
+element is the ayah number (also known as a verse number), and the second element is the
+content of the ayah.
 
-Each JSON file represents a surah (also known as a chapter). The format of the JSON
+See [Surah Al-Fatihah (English)](share/quran-json/TheQuran/en/1.json) for an example.
-files can be described as an array where the first element is an object that contains
-information about a surah, and the rest of the array is made up of two-element arrays.
-The first element is the ayah number (also known as a verse number), and the second
-element is the contents of the ayah. See Surah [Al-Fatihah](src/json/en/1.json) as
-an example.
 
-## <a id='srcsql-directory'>src/sql/</a>
+## <a id='bin/quran-json'>bin/quran-json</a>
 
-* [src/sql/schema.sql](src/sql/schema.sql) defines the schema of the database.
+The [bin/quran-json](bin/quran-json) executable is a utility for downloading
-* [src/sql/seed.sql](src/sql/seed.sql) can be used to populate a SQL database.
+the content of The Quran in multiple languages.
-* [src/sql/queries/](src/sql/queries) contains example SQL queries.
 
-## <a id='bin-directory'>bin/</a>
+### Usage
 
-The [bin/](bin/) directory contains scripts that generate the contents of the
+    Usage: quran-json pull [OPTIONS]
-[src/](src/) directory:
+        -l, --locale LOCALE        ar, en, pt, or fa (default: en)
-
-* JSON scripts
-  * [bin/json/pull-arabic](bin/json/pull-arabic) <br>
-    This script populates [src/json/ar/](src/json/ar/).
-  * [bin/json/pull-english](bin/json/pull-english) <br>
-    This script populates [src/json/en/](src/json/en/).
-  * [bin/json/pull-farsi](bin/json/pull-farsi) <br>
-    This script populates [src/json/fa/](src/json/fa/).
-  * [bin/json/pull-portuguese](bin/json/pull-portuguese) <br>
-    This script populates [src/json/pt/](src/json/pt/).
-
-* SQL scripts
-  * [bin/sql/create-sql-seed-file](bin/sql/create-sql-seed-file) <br>
-    This script creates [src/sql/seed.sql](src/sql/seed.sql).
 
 ## Thanks
 
 First and foremost, Alhamdulillah.
 
 I'd also like to extend thanks to the following websites for providing
-the content that quran-pull downloads:
+the content that quran-json downloads:
 
   * https://searchtruth.com for the original Arabic.
   * https://quran.com for the English, Portuguese, and Farsi translations.

bin/quran-json

@@ -16,8 +16,8 @@ end
 # main
 def main(argv)
   root_dir = File.realpath(File.join(__dir__, ".."))
-  lib_dir = File.join(root_dir, "lib", "quran-pull")
+  lib_dir = File.join(root_dir, "lib", "quran-json")
-  libexec_dir = File.join(root_dir, "libexec", "quran-pull")
+  libexec_dir = File.join(root_dir, "libexec", "quran-json")
   require File.join(lib_dir, "pull")
 
   case ARGV[0]
@@ -29,7 +29,7 @@ def main(argv)
       wait spawn(libexec_dir, "searchtruth.com", *argv[1..])
     end
   else
-    warn "Usage: quran-pull pull|help [OPTIONS]"
+    warn "Usage: quran-json pull [OPTIONS]"
   end
 end
 main(ARGV)

lib/quran-json/command.rb

@@ -0,0 +1,37 @@
+module Command
+  require "ryo"
+  require "json"
+  require "io/line"
+
+  def root_dir
+    File.realpath File.join(__dir__, "..", "..")
+  end
+
+  def share_dir
+    File.join(root_dir, "share", "quran-json")
+  end
+
+  def data_dir
+    File.join(share_dir, "data")
+  end
+
+  def quran_dir
+    File.join(share_dir, "TheQuran")
+  end
+
+  def line
+    @line ||= IO::Line.new($stdout)
+  end
+
+  def count
+    @count ||= Ryo.from(
+      JSON.parse File.binread(File.join(data_dir, "count.json"))
+    )
+  end
+
+  def surah_info
+    @surah_info ||= Ryo.from(
+      JSON.parse File.binread(File.join(data_dir, "surahinfo.json"))
+    )
+  end
+end

lib/quran-json/pull.rb

@@ -0,0 +1,89 @@
+class Pull
+  require "ryo"
+  require "json"
+  require "net/http"
+  require "fileutils"
+  require "optparse"
+  require_relative "command"
+  include Command
+  include FileUtils
+
+  attr_reader :options,
+              :source,
+              :http
+
+  def self.cli(argv)
+    op = nil
+    options = Ryo({locale: "en"})
+    OptionParser.new(nil, 26, " " * 2) do |o|
+      o.banner = "Usage: quran-json pull [OPTIONS]"
+      op = o
+      o.on("-l", "--locale LOCALE", "ar, en, pt, or fa (default: en)")
+    end.parse(argv, into: options)
+    options
+  rescue
+    puts op.help
+    exit
+  end
+
+  def initialize(options)
+    @options = options
+    @source = sources[options.locale]
+    @http = Net::HTTP.new(source.http.hostname, 443).tap { _1.use_ssl = true }
+  end
+
+  def pull_surah(surah_no, &b)
+    pull req_path(vars(binding)), &b
+  end
+
+  def pull_ayah(surah_no, ayah_no, &b)
+    pull req_path(vars(binding)), &b
+  end
+
+  def write(surah_no, rows)
+    dir = File.join(quran_dir, options.locale)
+    mkdir_p(dir)
+    rows.unshift(Ryo.table_of(surah_info[surah_no - 1]))
+    File.binwrite File.join(dir, "#{surah_no}.json"), JSON.pretty_generate(rows)
+  end
+
+  def keepalive
+    http.start
+    yield
+  ensure
+    http.finish
+  end
+
+  def exist?(surah_no)
+    File.exist? File.join(quran_dir, options.locale, "#{surah_no}.json")
+  end
+
+  private
+
+  def req_path(vars)
+    format source.http.path, source.http.vars.map { [_1.to_sym, vars[_1.to_sym]] }.to_h
+  end
+
+  def pull(path)
+    res = http.get(path)
+    case res
+    when Net::HTTPOK
+      yield(res)
+    else
+      ##
+      # TODO: Handle error
+    end
+  end
+
+  def vars(binding)
+    binding.local_variables.map do
+      [_1.to_sym, binding.local_variable_get(_1)]
+    end.to_h
+  end
+
+  def sources
+    @sources ||= Ryo.from(
+      JSON.parse File.binread(File.join(data_dir, "sources.json"))
+    )
+  end
+end

libexec/quran-json/quran.com

@@ -0,0 +1,32 @@
+#!/usr/bin/env ruby
+lib_dir = File.realpath File.join(__dir__, "..", "..", "lib", "quran-json")
+require File.join(lib_dir, "pull")
+require "optparse"
+require "nokogiri"
+
+def grep(res)
+  html = Nokogiri::HTML(res.body)
+  el = html.css("div[class^='TranslationText']").last
+  el.text.gsub(/[0-9]/, "")
+end
+
+##
+# main
+def main(argv)
+  cmd = Pull.new(Pull.cli(argv))
+  cmd.keepalive do
+    1.upto(114) do |surah_no|
+      next if cmd.exist?(surah_no)
+      rows = []
+      1.upto(cmd.count[surah_no]) do |ayah_no|
+        cmd.pull_ayah(surah_no, ayah_no) do |res|
+          rows.concat([ayah_no, grep(res)])
+        end
+        cmd.line.rewind.print "Surah #{surah_no} [#{ayah_no}/#{cmd.count[surah_no]}]"
+      end
+      cmd.write(surah_no, rows)
+      cmd.line.end
+    end
+  end
+end
+main(ARGV)

libexec/quran-json/searchtruth.com

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+lib_dir = File.realpath File.join(__dir__, "..", "..", "lib", "quran-json")
+require File.join(lib_dir, "pull")
+require "optparse"
+require "nokogiri"
+
+def grep(res)
+  html = Nokogiri::HTML(res.body)
+  html.css("table[dir='rtl'] tr td div:last-child").map { _1.text.strip }
+end
+
+##
+# main
+def main(argv)
+  cmd = Pull.new(Pull.cli(argv))
+  cmd.keepalive do
+    1.upto(114) do |surah_no|
+      next is cmd.exist?(surah_no)
+      rows = []
+      cmd.pull_surah(surah_no) do |res|
+        rows.concat(grep(res).map.with_index(1) { [_2, _1] })
+      end
+      cmd.line.rewind.print "Surah #{surah_no} [#{surah_no}/114]"
+      cmd.write(surah_no, rows)
+    end
+    cmd.line.end
+  end
+end
+main(ARGV)