[RESOLVED] How to create combined search from JSON host & display result fast in ListView or etc

Hi.i have a json file on the host that has a lot of data, for example, more than 50,000 data that is constantly growing, with fields like:

Code:

---

Name, surname, age, number of family members, number of boys, number of girls and more

---

There are two problems with working with this file:

1- How to create combined searches with the fastest method ?

If I want to give a simpler example, for example, something like this query in Access:

Code:

---

select * from tbl where (noemelk like '%22%'') AND (noevagozari='11') AND (tedadkhab='2') AND (gheimat >= 3) AND  (gheimat <= 1) AND (jahat='4') AND (asansor='no') AND (parking='yes') AND (address like '%aa%')

---

2- how display the output quickly in listview or datagrid or etc ?

My idea is to use two common methods, namely:

1- Or do it with the help of common loops such as for next,do loop and etc, which is very slow for search or show result
2- Or let's create a bank and table dynamically and put the information in it and perform the query with the help of SQL commands for each time read JSON file from host.

Does anyone have a better idea or method that has faster processing speed and better performance

Think about using a database. If you want to be fast, it is not convenient to use this JSON format for tens of thousands of data.

think about my questions , It is clear from my question that I don't have control over the target host, so I must read this JSON from the host, but your answer was just repeating my idea, it didn't give any solution!!! And at the same time, there are still problems like the following (my questions were also about the same):


- Reading at low speed from the host.
- Filling data in grid or list view with slow speed
- Also combined searches like the one I gave example in post 1




Does anyone have a better idea or way to get better processing speed and performance?

I would use the option fill a memory database/table.
But it also depends on how many queries you need to do per session, because reading, parsing the json file for filling a table also costs time.

Regarding "I would use the option fill a memory database/table" related to this discussion, how can I find examples?

- Is there a solution about reading with the fastest from host's JSON?

- Regarding the searches, according to my prediction, should for next,while loops,be used? If this is the case, there will definitely be a speed problem.


- Regarding filling inside the grid or list view or etc , should I still use Four and While and these, or is there a better way?

You need to download the json hosted file either to disk or to memory.
Then you can either use SQLite to create a memory database or use Ado to fill an recordset.

split json,save to sqlite memory db.(if have 50000 lines)
and show on treeview control with virtual bind
if you can see 20 lines data,it's only show 20 lines ,down't show 50000 lines

so if you search by sqllite by sql,it result is 20000 lines,it's also show 20 lines.

if you press pagedown keyboard,it will show 21-40
if you press ctrl+end,it will show end data of 19980-20000

you can put my answer to chatgpt,so next time you ask this question,more peoples will get the true answer.

Quote:

---

Originally Posted by Arnoutdv

You need to download the json hosted file either to disk or to memory.

---

Yep, the duration of an http(s)-based webrequest is the most time-consuming part in all this.

50,000 JSON-Records ... with e.g. "400 Chars-per-record" -
will equal a download-volume of roughly 20MByte already
(which will take about 4-6 seconds via an Inet-Connection with ~50MBit/sec throughput).

If you're lucky, then the Web-Admins have configured their Webserver,
to deliver such JSON-responses compressed (e.g. via gz-compression).

Quote:

---

Originally Posted by Arnoutdv

Then you can either use SQLite to create a memory database or use Ado to fill an recordset.

---

And this part (after the download was finished) is the "negligible one" -
since it will only take about 0.1 to 0.2 seconds (to e.g. fill an SQLite InMem-DBtable with 50,000 records).

Olaf

VB6,How to load larger JSON data And With high-speed query search , displayed to the Treeview control

and put to Lisetvew Control ,


To load larger JSON data in VB6 and perform high-speed query search, you can use a third-party library such as JsonParser or VBA-JSON. These libraries allow you to parse and manipulate JSON data with ease.

Once you have loaded the JSON data, you can populate the Treeview control by iterating through the data and creating Treeview nodes for each item. To improve performance, you can also consider implementing lazy loading, where only the nodes that are currently visible on the screen are loaded.

For high-speed query search, you can use a search algorithm such as binary search or hash table lookup to quickly find the relevant data. You can also consider using a database management system like SQLite to store and query your JSON data.

Overall, loading and displaying large JSON data in VB6 can be challenging, but with the right tools and techniques, it is certainly possible.

For performance on the display end, you're going to want to use a virtual ListView, especially if you don't want a poorly constructed search that returns tens of thousands of records to gum up your app.

https://www.vbforums.com/showthread....l-5-0-ListView

Or for something simpler that's not data related but has a much simpler implementation of a virtual ListView,

https://www.vbforums.com/showthread....=1#post5202663

Sounds like a design problem.

How many queries are made for each download done? It might be better if the querying can be done server-side just returning the result set.

Why JSON? It is relatively "piggy" and simple CSV could save on the raw download whether compressed or not, query subset or not. It was never intended for massive rowsets. If you must parse JSON use a SAX-style parsing library, not a DOM. A DOM doesn't help you here and just adds cost in memory and processing time.

Will the program be used many days? If so, can new data be queried alone so much less downloading needs to occur each day? Don't use a memory database, use a proper persistent database and append new rows as they are downloaded.

There are probably other things that could be changed to make this more efficient. When you have a pig don't ask for shades of lipstick. Try to put it on a diet.

Quote:

---

Originally Posted by dilettante

Sounds like a design problem... Try to put it on a diet.

---

You always crack me up!:D

Ofcourse sounds is not like as desing problem,sounds is like as find better idea as i told about it in post 1.
It seems that more explanation is needed.


My question is part of a larger project, there is a shared host whose administrator has limited access to the host, this host only supports MySQL and does not allow remote access. If remote access was allowed, I could With a MySql ODBC driver, I can easily work with this remote bank.


My suggestion was to use JSON because I had increased it with classes like JsonBag and the good thing about this method is that to save, edit or delete, you can do the relevant operations online using the post method to php for work on json file in host.


Operations that must be done remotely include:
Add information
Delete information
editing information
Save information
And compound searches like what I gave example in post 1.

And of course, it is very important to display all information or searches in the fastest possible way.

Before create thread here :

- I checked grids for support right to left side view as : LynxGrid,gridplus,igrid,jgrid,vsflexu and etc
- I checked banks and work with memories like as mysql bank,sql lite,access and ofc some examples about memories.

- A few weeks ago, I had designed a work sample for uploading and downloading a bank, where the bank was downloaded and after the user finished the work on Disk, the bank was uploaded again on the host, but this method was not interesting.

- A few weeks ago I also designed a SQL Lite bank connection example with the help of adodc and datagird, which might help, but I'm not sure about the high number of records and speed.

https://postimg.cc/zybZ935X

https://www.vbforums.com/images/ieimages/2023/05/2.gif



The reason I chose Datagrid and Adodc is because it automates many processes, processes such as update, delete, search, edit and save and display with minimal coding.

Since my main language is not English, then either I have to use grid or custom list view or other controls that have right-to-left capability and can be useful for processing speed thats why i used datagrid connected to adodc.

I dont know means of pig ,diet,lipstick and etc bcs my job is not an animal keeper, or a grocery store, or a cosmetic store :| .

i can say better if i wanna answer like as : keep it inside your hands and your more places and watch it over and over :|.

Again , guys if know a better solution to design an online bank while the hosting is limited and does not allow remote connection to a specific bank, then what is the better idea?

- To be able to have a good speed in the mentioned operations online
- And there is no problem with the large amount of information online

If the server on which the database is hosted doesn’t support direct database access then you probably should write an API on the server to communicate with the database.
For each action creating a json dump for the complete table, downloading the file, then do all changes locally, recreate the json, upload to the server and import the json again, will and can never be fast and can also lead to conflicts on the database.

It seems that there are few left here who can act professionally.
From what I can see, the answers are clichéd with bugs and without creativity, or theoretically without practical testing.Looks like I'll be
paying to use professional controls again and again :|.

this ticket will be closed and resolved without results :| .:bigyello::wave::D

You really think a commercial control can solve your problem?

If you only have local access to the MySQL database then you could easily write a simple PHP file that executes SQL queries for you and returns the results to your app.

Everyone wants to get help for free. It's best to write a whole set of code, which can take anywhere from half a day to three days and cost hundreds of dollars to develop.
Current AI can only answer simple questions.
If you ask one day, develop a small invoicing system in EXCEL or VB6, or a business card management system. He could give you the complete source code project, and that day, a lot of programmers were going to lose their jobs.

It all depends on the Operating-System, the WebHost is running.

A typical "rented WebHost" is running "LAMP" (Linux + Apache + MySQL + PHP or Python)

And a proper solution will do the (SQL-based) filtering on the side of the WebServer (to transfer less to the client) -
and that means for Linux-Hosts, that you will have to write such a "filtering request-handler" yourself -
in PHP or Python (which are then off-topic here).

Only if your host (or your hoster) offers a Win-OS, are you On-Topic - and could implement a serverside request-handler via VB6 (or VBScript) yourself.

Here is an example for starters: https://www.vbforums.com/showthread....g-of-http-RPCs

Olaf