-
Input Needed, on Multiple Levels
There's a problem that I'm involved with, and I'm looking for suggestions.
We get information from a small-ish number of people (roughly 100-300 per year). We have no control over them other than to require that they give us this information. The information is a single table with a variable number of rows from 0 to potentially thousands. In the past, what was done was giving them a spreadsheet template and asking them to fill that out. Such a solution is still possible, but I'm looking for something better. The issue with the spreadsheet is that we have to give them the template, or else we are sure to get back all kinds of weird and misfit data in all kinds of strange formats. If we gave them a web form to fill out, that would be insanely tedious, since they will already have the data in some kind of electronic form (or at least most of them do, though we have no say or idea what format that will be). What would be ideal would be essentially a web based spreadsheet into which they could copy and paste from an existing spreadsheet, but where we could set the format for the destination both in terms of data type, columns, and requirements (what is mandatory and what is optional). I've seen one control like that, from DevExpress, but it has some licensing limitations that aren't going to work (my understanding is that each end user has to have a license, which is unlike any of their other controls).
The general goal is to get a single table of tabular data with an unknown number of rows, where we can minimize the pain for the end user to get the data to us, while letting us set which columns exist, which columns are mandatory vs optional, and the data types for the columns.
Any suggestions?
-
Re: Input Needed, on Multiple Levels
We are using telerik web controls in one of the companies. I think the end license is per creator but it is possible to download the setup and run it in as many PC's we want.
Although we only develop in 2 users PCs. What I mean is the development is licensed. We are building this into a company private network website and users can log in and input their data. The number of users login are for 30 to 60 and the server is a simple HP one, not sure about the exact model but i can look it up. I think it's xeon from 2016. I'm not sure if this is what you want but I'm posting an image. Sorry for the wipes but this is company disclosure data so I can't show that online but you will get the idea. If you can't view it let me know I'll try to attach it to my files. Basically the part you want is the last one that has a " + " and says add new record and it opens the spreadsheet.
Attachment 181922
-
Re: Input Needed, on Multiple Levels
Well, you certainly wiped it effectively, since I can't see the attachment at all.
This is not going to be internal, which is one of the things that is different for me. I have only written programs for internal use, and generally desktop. In this case, neither one is an option. It has to be web and it has to be public facing. I'll take a look at telerik. Seems like people have pretty mixed views on those, around here.
-
1 Attachment(s)
Re: Input Needed, on Multiple Levels
It is used internally from the company as we block the external connections, else it is good to go on the web.
Also I don't say that telerik are super wow, I just say we are using those so they are at least tested.
Let me try the picture again...
Attachment 181924
-
Re: Input Needed, on Multiple Levels
-
Re: Input Needed, on Multiple Levels
It seems a control that could handle data that has no predefined structure and organized the data in a specific format when it's copy/pasted would be a big ask. Or are you wanting a control that allows the user to copy/paste and then you would organized the data.
How about a web site with a downloadable template and then periodic autogenerated emails/messages as a reminders, with instructions????
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
jdc2000
I've thought about that, though only a little. I believe we will got to Office 365, but have not done so yet. It would have some appeal, so I'll have to look at that route in more detail.
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
wes4dbt
It seems a control that could handle data that has no predefined structure and organized the data in a specific format when it's copy/pasted would be a big ask. Or are you wanting a control that allows the user to copy/paste and then you would organized the data.
How about a web site with a downloadable template and then periodic autogenerated emails/messages as a reminders, with instructions????
Yeah, it's a big ask. Getting correctly formatted data from people you have some control over is a big ask. Getting it from random cats? Well, that's unlikely. That's pretty much why I'm asking here. What we are looking to do has so many possible failure points that I wanted more minds providing input.
-
Re: Input Needed, on Multiple Levels
I can imagine this is daunting.
You probably have to deal with Auntie Em in Patootie County who works two days a week on a machine her nephew Walwart set up with H4x0r Linux and a split Dvorak keyboard as well as Drumstick Poindexter in Nether Wallop who wants to export his data from DB2.
-
Re: Input Needed, on Multiple Levels
Quote:
We get information from a small-ish number of people (roughly 100-300 per year). We have no control over them other than to require that they give us this information.
Some 'left field' questions:
Why do you require they provide the info? Is this government mandated? What law mandates that you are required to obtain this data? Is it just for your analysis? Does the provider get anything out of this? What happens/repercussions if they don't provide what is asked or only provide partial or incorrect data?
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
dilettante
I can imagine this is daunting.
You probably have to deal with Auntie Em in Patootie County who works two days a week on a machine her nephew Walwart set up with H4x0r Linux and a split Dvorak keyboard as well as Drumstick Poindexter in Nether Wallop who wants to export his data from DB2.
Oh I WISH it was that easy. Instead, we get to deal with a pretty awesome, and VERY dedicated guy who is in his 90s, has never used a computer, and never will. The solution doesn't have to work for him, though. We'll make accommodations for him, as he's worth it, but some of the other folks are going to be challenging.
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
2kaud
Some 'left field' questions:
Why do you require they provide the info? Is this government mandated? What law mandates that you are required to obtain this data? Is it just for your analysis? Does the provider get anything out of this? What happens/repercussions if they don't provide what is asked or only provide partial or incorrect data?
It is the culmination of a contract that they provide some minimal data. I could probably cite the statute that requires this, but I don't know it off the top of my head. What happens if the data isn't received is that the contract isn't renewed the following year, which is usually important to the people. However, there's a whole lot of leeway in the system, despite there being what appears to be fairly rigid rules. Therefore, we're trying to get to something none too onerous for either end of the transaction.
-
Re: Input Needed, on Multiple Levels
Quote:
they will already have the data in some kind of electronic form
Can't you include in the contract that this data has to be provided in specified digital formats automatically? ie csv, json, xml et al Surely having the info automatically provided digitally is better than having data manually entered...
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
2kaud
Can't you include in the contract that this data has to be provided in specified digital formats automatically? ie csv, json, xml et al Surely having the info automatically provided digitally is better than having data manually entered...
This process must be automated.
-
Re: Input Needed, on Multiple Levels
Yeah, that's true.
They'll have the data we want, and a LOT more that we can't even deal with. Essentially, they won't be giving us all the data they have, they'll be extracting some subset of the data they have, which is all we can use. We have no say over how they hold and maintain their data, just on the small subset of that data that they have to give to us. So, I'm looking for a way to make it fairly easy for them to get just that which we need.
-
Re: Input Needed, on Multiple Levels
Based on what was described in the OP, I would have designed a program just for that. Could be a web application or a plain old desktop app. People don't like change but what I've observed is that once people give something new a try, they tend to eventually fall in comfortably with it. The problem here is getting them past that initial comfort. Of course this is just my opinion. I can't say whether this is right or wrong. I like making things as easy as possible for people but at the same time I don't like headaches and this sounds like a lot of headache trying to find a solution that makes everybody on both sides happy.
-
Re: Input Needed, on Multiple Levels
Perhaps, but in this case, we have very few options for the other side. We can't ask them to install anything, so a desktop app is out of the question. Asking them to re-enter all their data into some web form would be clearly unappealing to them. We CAN require that they do that, but the data won't be as good if we do.
Keep in mind, that what they are providing is a subset of data they already collected for their own purposes, so they'll already have the data in whatever storage format they find suitable. We're asking for some subset of that data. That's an unusual situation for the web.
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
Shaggy Hiker
Perhaps, but in this case, we have very few options for the other side. We can't ask them to install anything, so a desktop app is out of the question. Asking them to re-enter all their data into some web form would be clearly unappealing to them. We CAN require that they do that, but the data won't be as good if we do.
Keep in mind, that what they are providing is a subset of data they already collected for their own purposes, so they'll already have the data in whatever storage format they find suitable. We're asking for some subset of that data. That's an unusual situation for the web.
If you need to be vague, that's fine. But what isn't adding up is why a spreadsheet template is easy and a web form is out of the question, when in both cases, they are presumably taking the data from [Their App] and (copying/pasting?) the data into (your solution).
You are basically asking us to help you make a process easier, when no one here has a clue what is happening on either end of said process.
That being said, my suggestion is to give them multiple options. Give them the spreadsheet template and give them a web form, and maybe other options. Basically, you seem to be deciding for them what they will view as tedious. Maybe they don't feel the same way. Maybe some of these entities are savvy enough to build a bot that fills out the web form for them with whatever data they have. Who knows.
Good luck.
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
Shaggy Hiker
so they'll already have the data in whatever storage format they find suitable. We're asking for some subset of that data.
If this is a standard format that they are consistent with, an application tailored to that specific format could work. They won't have to do anything special and you guys could automate it on your side. This of course depends completely on them being consistent with their format every single time they send the data.
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
OptionBase1
If you need to be vague, that's fine. But what isn't adding up is why a spreadsheet template is easy and a web form is out of the question, when in both cases, they are presumably taking the data from [Their App] and (copying/pasting?) the data into (your solution).
You are basically asking us to help you make a process easier, when no one here has a clue what is happening on either end of said process.
That being said, my suggestion is to give them multiple options. Give them the spreadsheet template and give them a web form, and maybe other options. Basically, you seem to be deciding for them what they will view as tedious. Maybe they don't feel the same way. Maybe some of these entities are savvy enough to build a bot that fills out the web form for them with whatever data they have. Who knows.
Good luck.
I'm not trying to be vague. Technically, we haven't quite nailed down WHAT data we are going to get from them, but it has been done via emailed spreadsheets in the past, so I know that will work...one way or another.
We have no clue what is happening on their end. This is biological survey data, of which we will get some subset including species numbers and about one to two dozen other fields. They could be doing darn near anything (including making stuff up, but we hope not). I know that at least one person is just scribbling on a piece of paper, and another is a multinational group that sends us a thousand page book every year covering everything they did around the world, even though we can't possibly make use of it. I have no idea why they do that. Perhaps they just want to get the word out about what they do, perhaps it's just easier, since they had the thing lying around. So, what are they doing for data collection? It could be a high-tech distributed operation, it could be scraps of paper, we have no idea.
I was originally thinking of a web form, but the only web forms I have ever seen require the user to type data into each field. We have those one to two dozen fields, but there could be anywhere from zero to several thousand lines (probably only a few hundred, on average). They won't be using our form to collect the data. They don't work for us, or with us. The data they are required to supply is in compliance with a permit. So, if we use a web form, my only experience with them would suggest that they'd be doing a LOT of typing, with all the typos that can come along with that. If there is a web form solution that allows the copying and pasting of large blocks of data, that would be great, but that's why I'm asking, because I don't know of anything like that.
At this point, the suggestion about MS 365 seems like it would work the easiest, but if there's a way to blast a large table into a web form in a set format, that would be what I'm looking for.
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
Niya
If this is a standard format that they are consistent with, an application tailored to that specific format could work. They won't have to do anything special and you guys could automate it on your side. This of course depends completely on them being consistent with their format every single time they send the data.
I think I answered this with the response to OB, but essentially: No. We are getting a subset of whatever it is that they are collecting. I doubt that anybody will be collecting JUST what we want, but every one of them will be collecting at LEAST what we require. They won't be collecting that because of us, either. As it turns out, there are certain things that just happen to ALWAYS be collected when it comes to biological data, which amounts to who, what, when, where, and how many. We just require that subset. They might also be collecting genetic information, climate information, water chemistry, spatial distribution, physiological information, tag information, and on and on. So, their data collection will be based on whatever it is that they are interested in, and it won't fit our format. However, they will end up collecting the who, what, when, where, and how many, which we are looking for. It's just kind of how biological data works out.
-
Re: Input Needed, on Multiple Levels
Quote:
Originally Posted by
Shaggy Hiker
I'm not trying to be vague. Technically, we haven't quite nailed down WHAT data we are going to get from them, but it has been done via emailed spreadsheets in the past, so I know that will work...one way or another.
We have no clue what is happening on their end.
I thought possibly you couldn't reveal more specific details because of confidentiality of some sort.
With that out of the way, the last sentence that I quoted (and more in that paragraph that I didn't quote) is a bit of a red flag in my opinion. Certainly, the people using the paper data storage method need to re-type that same data into something (Excel template, web form, or other), so what that something is shouldn't matter much to them. But I don't see how you can be expected to streamline a process when you don't have insight into what is being done on the other end.
But that goes back to my previous suggestion. Give people options. They will migrate to the one that works best for them.
Quote:
Originally Posted by
Shaggy Hiker
I was originally thinking of a web form, but the only web forms I have ever seen require the user to type data into each field. We have those one to two dozen fields, but there could be anywhere from zero to several thousand lines (probably only a few hundred, on average). They won't be using our form to collect the data. They don't work for us, or with us. The data they are required to supply is in compliance with a permit. So, if we use a web form, my only experience with them would suggest that they'd be doing a LOT of typing, with all the typos that can come along with that. If there is a web form solution that allows the copying and pasting of large blocks of data, that would be great, but that's why I'm asking, because I don't know of anything like that.
Last bolded sentence??? Your replies in this thread (and mine, and everyone else's) are being made using a web form and a box that allows the copying and pasting of large blocks of data. This site may impose a post size limit on form submission, or specify a "maxlength", but those can be adjusted.
Have a few plain old single line fields for the boilerplate data (Name, Address, whatever), and then a big textarea where they can paste in all of the information that they think you want them to submit.
Good luck.
-
Re: Input Needed, on Multiple Levels
Using this plain HTML:
Code:
<HTML>
<HEAD>
<TITLE>TextArea Test page</TITLE>
</HEAD>
<BODY>
<FORM NAME="Test">
<TEXTAREA ID="TextArea1" NAME="TextArea1" ROWS=30 COLS=100>
</TEXTAREA>
</FORM>
</BODY>
</HTML>
Just copy that and save it to your desktop as an html file and open it in a browser.
For testing I copied and pasted some lines of text into the textarea on my browser until I had over 160,000 lines of text in it. It lagged a bit as it populated, but only for a second or two.
-
Re: Input Needed, on Multiple Levels
It seems like that would allow people to put anything in there with any kind of formatting. We don't expect people to get this right, or at least I don't, but only because I've done this too long. If you give person nothing more than a button to push, somebody will figure out how to push it wrong. In this case...we're headed for a mess, because we're looking for lots of fields and hoping that the people collecting the data have the information in a usable format. I'm not expecting to prevent that mess, but I AM going to try to push people into tidying the mess up a bit, so I need to impose some structure on what they paste.
Who am I kidding...this is going to be a morass, and I'll be doing no more than providing some sign posts.
