Results 1 to 19 of 19

Thread: [RESOLVED] How to create combined search from JSON host & display result fast in ListView or etc

  1. #1

    Thread Starter
    Fanatic Member Black_Storm's Avatar
    Join Date
    Sep 2007
    Location
    any where
    Posts
    575

    Resolved [RESOLVED] How to create combined search from JSON host & display result fast in ListView or etc

    Hi.i have a json file on the host that has a lot of data, for example, more than 50,000 data that is constantly growing, with fields like:

    Code:
    Name, surname, age, number of family members, number of boys, number of girls and more
    There are two problems with working with this file:

    1- How to create combined searches with the fastest method ?

    If I want to give a simpler example, for example, something like this query in Access:
    Code:
    select * from tbl where (noemelk like '%22%'') AND (noevagozari='11') AND (tedadkhab='2') AND (gheimat >= 3) AND  (gheimat <= 1) AND (jahat='4') AND (asansor='no') AND (parking='yes') AND (address like '%aa%')
    2- how display the output quickly in listview or datagrid or etc ?

    My idea is to use two common methods, namely:

    1- Or do it with the help of common loops such as for next,do loop and etc, which is very slow for search or show result
    2- Or let's create a bank and table dynamically and put the information in it and perform the query with the help of SQL commands for each time read JSON file from host.

    Does anyone have a better idea or method that has faster processing speed and better performance
    [ ... active on skype and discord ... ] ,[always strive to achieve your dreams] , [always try,dont stop,never say never]

  2. #2
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: How to create combined search from JSON host & display result fast in ListView or

    Think about using a database. If you want to be fast, it is not convenient to use this JSON format for tens of thousands of data.

  3. #3

    Thread Starter
    Fanatic Member Black_Storm's Avatar
    Join Date
    Sep 2007
    Location
    any where
    Posts
    575

    Re: How to create combined search from JSON host & display result fast in ListView or

    think about my questions , It is clear from my question that I don't have control over the target host, so I must read this JSON from the host, but your answer was just repeating my idea, it didn't give any solution!!! And at the same time, there are still problems like the following (my questions were also about the same):


    - Reading at low speed from the host.
    - Filling data in grid or list view with slow speed
    - Also combined searches like the one I gave example in post 1




    Does anyone have a better idea or way to get better processing speed and performance?
    Last edited by Black_Storm; May 30th, 2023 at 02:01 AM.
    [ ... active on skype and discord ... ] ,[always strive to achieve your dreams] , [always try,dont stop,never say never]

  4. #4
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: How to create combined search from JSON host & display result fast in ListView or

    I would use the option fill a memory database/table.
    But it also depends on how many queries you need to do per session, because reading, parsing the json file for filling a table also costs time.

  5. #5

    Thread Starter
    Fanatic Member Black_Storm's Avatar
    Join Date
    Sep 2007
    Location
    any where
    Posts
    575

    Re: How to create combined search from JSON host & display result fast in ListView or

    Regarding "I would use the option fill a memory database/table" related to this discussion, how can I find examples?

    - Is there a solution about reading with the fastest from host's JSON?

    - Regarding the searches, according to my prediction, should for next,while loops,be used? If this is the case, there will definitely be a speed problem.


    - Regarding filling inside the grid or list view or etc , should I still use Four and While and these, or is there a better way?
    [ ... active on skype and discord ... ] ,[always strive to achieve your dreams] , [always try,dont stop,never say never]

  6. #6
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: How to create combined search from JSON host & display result fast in ListView or

    You need to download the json hosted file either to disk or to memory.
    Then you can either use SQLite to create a memory database or use Ado to fill an recordset.

  7. #7
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: How to create combined search from JSON host & display result fast in ListView or

    split json,save to sqlite memory db.(if have 50000 lines)
    and show on treeview control with virtual bind
    if you can see 20 lines data,it's only show 20 lines ,down't show 50000 lines

    so if you search by sqllite by sql,it result is 20000 lines,it's also show 20 lines.

    if you press pagedown keyboard,it will show 21-40
    if you press ctrl+end,it will show end data of 19980-20000

    you can put my answer to chatgpt,so next time you ask this question,more peoples will get the true answer.

  8. #8
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: How to create combined search from JSON host & display result fast in ListView or

    Quote Originally Posted by Arnoutdv View Post
    You need to download the json hosted file either to disk or to memory.
    Yep, the duration of an http(s)-based webrequest is the most time-consuming part in all this.

    50,000 JSON-Records ... with e.g. "400 Chars-per-record" -
    will equal a download-volume of roughly 20MByte already
    (which will take about 4-6 seconds via an Inet-Connection with ~50MBit/sec throughput).

    If you're lucky, then the Web-Admins have configured their Webserver,
    to deliver such JSON-responses compressed (e.g. via gz-compression).

    Quote Originally Posted by Arnoutdv View Post
    Then you can either use SQLite to create a memory database or use Ado to fill an recordset.
    And this part (after the download was finished) is the "negligible one" -
    since it will only take about 0.1 to 0.2 seconds (to e.g. fill an SQLite InMem-DBtable with 50,000 records).

    Olaf

  9. #9
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: How to create combined search from JSON host & display result fast in ListView or

    VB6,How to load larger JSON data And With high-speed query search , displayed to the Treeview control

    and put to Lisetvew Control ,


    To load larger JSON data in VB6 and perform high-speed query search, you can use a third-party library such as JsonParser or VBA-JSON. These libraries allow you to parse and manipulate JSON data with ease.

    Once you have loaded the JSON data, you can populate the Treeview control by iterating through the data and creating Treeview nodes for each item. To improve performance, you can also consider implementing lazy loading, where only the nodes that are currently visible on the screen are loaded.

    For high-speed query search, you can use a search algorithm such as binary search or hash table lookup to quickly find the relevant data. You can also consider using a database management system like SQLite to store and query your JSON data.

    Overall, loading and displaying large JSON data in VB6 can be challenging, but with the right tools and techniques, it is certainly possible.

  10. #10
    PowerPoster
    Join Date
    Jul 2010
    Location
    NYC
    Posts
    5,651

    Re: How to create combined search from JSON host & display result fast in ListView or

    For performance on the display end, you're going to want to use a virtual ListView, especially if you don't want a poorly constructed search that returns tens of thousands of records to gum up your app.

    https://www.vbforums.com/showthread....l-5-0-ListView

    Or for something simpler that's not data related but has a much simpler implementation of a virtual ListView,

    https://www.vbforums.com/showthread....=1#post5202663

  11. #11
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How to create combined search from JSON host & display result fast in ListView or

    Sounds like a design problem.

    How many queries are made for each download done? It might be better if the querying can be done server-side just returning the result set.

    Why JSON? It is relatively "piggy" and simple CSV could save on the raw download whether compressed or not, query subset or not. It was never intended for massive rowsets. If you must parse JSON use a SAX-style parsing library, not a DOM. A DOM doesn't help you here and just adds cost in memory and processing time.

    Will the program be used many days? If so, can new data be queried alone so much less downloading needs to occur each day? Don't use a memory database, use a proper persistent database and append new rows as they are downloaded.

    There are probably other things that could be changed to make this more efficient. When you have a pig don't ask for shades of lipstick. Try to put it on a diet.

  12. #12
    Addicted Member
    Join Date
    Feb 2022
    Posts
    167

    Re: How to create combined search from JSON host & display result fast in ListView or

    Quote Originally Posted by dilettante View Post
    Sounds like a design problem... Try to put it on a diet.
    You always crack me up!

  13. #13

    Thread Starter
    Fanatic Member Black_Storm's Avatar
    Join Date
    Sep 2007
    Location
    any where
    Posts
    575

    Re: How to create combined search from JSON host & display result fast in ListView or

    Ofcourse sounds is not like as desing problem,sounds is like as find better idea as i told about it in post 1.
    It seems that more explanation is needed.


    My question is part of a larger project, there is a shared host whose administrator has limited access to the host, this host only supports MySQL and does not allow remote access. If remote access was allowed, I could With a MySql ODBC driver, I can easily work with this remote bank.


    My suggestion was to use JSON because I had increased it with classes like JsonBag and the good thing about this method is that to save, edit or delete, you can do the relevant operations online using the post method to php for work on json file in host.


    Operations that must be done remotely include:
    Add information
    Delete information
    editing information
    Save information
    And compound searches like what I gave example in post 1.

    And of course, it is very important to display all information or searches in the fastest possible way.

    Before create thread here :

    - I checked grids for support right to left side view as : LynxGrid,gridplus,igrid,jgrid,vsflexu and etc
    - I checked banks and work with memories like as mysql bank,sql lite,access and ofc some examples about memories.

    - A few weeks ago, I had designed a work sample for uploading and downloading a bank, where the bank was downloaded and after the user finished the work on Disk, the bank was uploaded again on the host, but this method was not interesting.

    - A few weeks ago I also designed a SQL Lite bank connection example with the help of adodc and datagird, which might help, but I'm not sure about the high number of records and speed.

    https://postimg.cc/zybZ935X





    The reason I chose Datagrid and Adodc is because it automates many processes, processes such as update, delete, search, edit and save and display with minimal coding.

    Since my main language is not English, then either I have to use grid or custom list view or other controls that have right-to-left capability and can be useful for processing speed thats why i used datagrid connected to adodc.

    I dont know means of pig ,diet,lipstick and etc bcs my job is not an animal keeper, or a grocery store, or a cosmetic store :| .

    i can say better if i wanna answer like as : keep it inside your hands and your more places and watch it over and over :|.

    Again , guys if know a better solution to design an online bank while the hosting is limited and does not allow remote connection to a specific bank, then what is the better idea?

    - To be able to have a good speed in the mentioned operations online
    - And there is no problem with the large amount of information online
    Last edited by Black_Storm; May 31st, 2023 at 01:28 AM.
    [ ... active on skype and discord ... ] ,[always strive to achieve your dreams] , [always try,dont stop,never say never]

  14. #14
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: How to create combined search from JSON host & display result fast in ListView or

    If the server on which the database is hosted doesn’t support direct database access then you probably should write an API on the server to communicate with the database.
    For each action creating a json dump for the complete table, downloading the file, then do all changes locally, recreate the json, upload to the server and import the json again, will and can never be fast and can also lead to conflicts on the database.

  15. #15

    Thread Starter
    Fanatic Member Black_Storm's Avatar
    Join Date
    Sep 2007
    Location
    any where
    Posts
    575

    Re: How to create combined search from JSON host & display result fast in ListView or

    It seems that there are few left here who can act professionally.
    From what I can see, the answers are clichéd with bugs and without creativity, or theoretically without practical testing.Looks like I'll be
    paying to use professional controls again and again :|.

    this ticket will be closed and resolved without results :| .
    [ ... active on skype and discord ... ] ,[always strive to achieve your dreams] , [always try,dont stop,never say never]

  16. #16
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: [RESOLVED] How to create combined search from JSON host & display result fast in

    You really think a commercial control can solve your problem?

  17. #17
    Frenzied Member VanGoghGaming's Avatar
    Join Date
    Jan 2020
    Location
    Eve Online - Mining, Missions & Market Trading!
    Posts
    1,324

    Re: [RESOLVED] How to create combined search from JSON host & display result fast in

    If you only have local access to the MySQL database then you could easily write a simple PHP file that executes SQL queries for you and returns the results to your app.

  18. #18
    PowerPoster
    Join Date
    Jan 2020
    Posts
    3,746

    Re: [RESOLVED] How to create combined search from JSON host & display result fast in

    Everyone wants to get help for free. It's best to write a whole set of code, which can take anywhere from half a day to three days and cost hundreds of dollars to develop.
    Current AI can only answer simple questions.
    If you ask one day, develop a small invoicing system in EXCEL or VB6, or a business card management system. He could give you the complete source code project, and that day, a lot of programmers were going to lose their jobs.

  19. #19
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,219

    Re: [RESOLVED] How to create combined search from JSON host & display result fast in

    It all depends on the Operating-System, the WebHost is running.

    A typical "rented WebHost" is running "LAMP" (Linux + Apache + MySQL + PHP or Python)

    And a proper solution will do the (SQL-based) filtering on the side of the WebServer (to transfer less to the client) -
    and that means for Linux-Hosts, that you will have to write such a "filtering request-handler" yourself -
    in PHP or Python (which are then off-topic here).

    Only if your host (or your hoster) offers a Win-OS, are you On-Topic - and could implement a serverside request-handler via VB6 (or VBScript) yourself.

    Here is an example for starters: https://www.vbforums.com/showthread....g-of-http-RPCs

    Olaf

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width