Jump to content
RockWare Support Forum
Schelhorn19

Importing large XYZ Dataset

Recommended Posts

Hey ,

I just started working with RockWorks. I want to do a hydrogeological model. Therefor I have a textfile with 3 columns with the X-Y coordinates and the elevation Z( UTM degree) . In total I have 136 000 rows in the textfile. I have problems importing this whole dataset. RockWorks takes forever importing this data and at the end it just stops. If I only import 1000 rows, there is no problem. No errors and if I double check the data in the earthapps, it is at the desired coordinates and i can work with it.

Do you have an idea what is my problem? I think it should be possible for RockWorks to handle this amount of data.

I attached an example of my dataset.

Thank you for your help

Greetings

Josch

zellingen_coord_degree.txt

Share this post


Link to post
Share on other sites

Schelhorn,

I took a look at your file and I noticed a few things. First question I have is if you are importing this into the database or into the Utilities data sheet?

Your data is in this format:

"d"; "xcoord_UTM[degree]"; "ycoord_UTM[degree]"; "elevation[m]

"1";1.2345 ; 23.4567 ; 123

Your data is mixed with quoted text and float values. Ideally, you don't need to have quotes in your text. Having the quotes may cause issues if you want to import the data into the database. Cleaning this data up ahead of time is a good idea if you can. There's no need for quotes in the header or the names.

When you are importing the text into the datasheet, be sure to select the delimiter (the second option) as a semi-colon ( ;).

I suspect that one of these things is what was causing the import to hang up because I didn't have any problem importing the data. Be sure to use decluster when you make your maps because this could help keep it from hanging up.

I opened the file in Notepad and Excel and only found 1000 data points, so you must have lost some data when you exported it.

On a side note, your data are in Latitude/Longitude and not in UTM. Please be careful with using multiple coordinate systems and try to be consistent whenever possible.

Share this post


Link to post
Share on other sites

Hey,

thanks for your help. I cleaned up my textfile by deleating all quotes and unessary information. The new textfile I attached has 80 000 rows and 3 columns with coordinates in long lat and the elevation. (the whole dataset is too big to upload)

When creating a new projekt I choose the default folder settings and for the Project coordinates I choose "Local (Custom) longitude latitude in Meters". Is that right? I am wondering because my data is in degree.

In the Utitlities Data set I define the columns "x or y coordinates, decimal Degrees". Ok? The scan of the Datasheet works fine.

I want to load this data in the Utilities data sheet to create a 2D and 3D Grid-based Map based on xyz data. I use the method decluster.

For a smaller amount of rows like 10 000 rows , I have no problems to create the models, even though it is quite slowly.

For the whole data set RockWorks calculates about 1.5 h and at the end there is no result shown.

Is the data set just too big because its from a DEM with a 50 m resolution? What else could be my problem?

Thank you for your help.

Greeting Josch

zellingen_coord_degree80000.zip

Share this post


Link to post
Share on other sites

This is a pretty large data set so you will want to load the file to make the grid instead of using the data sheet.

I would stay away from using Local Coordinates, The maps you make here are small scale and UTM is much better for that.

Here's what I did to make a map from your data:

  • Be sure you make your project in UTM zone 32 (I used x/y and z meters)
  • load your data to the datasheet
  • convert the Lat/lon to UTM zone 32
  • You can scan the datasheet to get your model dimensions at this point based on the UTM
  • export the data to a text file
  • Load the text file using Map-> Gridbased map-> based on xyz
  • Because you have such a huge data set I used triangulation and got a good result

Share this post


Link to post
Share on other sites

hey Brian,

thanks for your answer and support.

I followed your instructions but it still doesn't work out as I wish. I could generate a surface modell of my data but it took me more than 3 h. Analizing it in RockPlot3D it was also very slow.

I know I have a huge dataset with a very high elevation resolution but I think RockWorks is developped to handle big projects. Is there any trick to fasten it up?

I have some some basic question for which I can't find an answer in the help and the tutorials.

My project idea is: I have a catchment with about 200km2 for which I want to do a complex model which includes

- surface model

- a subsurface model

- stratigraphy

- lithology

- the aquifers

-boreholes

- faults

- and so on

My database consists out of

- DEM 50 m resolution (XYZ file with 3 rows and 136 000 rows)

- about 100 borehole data

- 4 crosssections with the stratigraphy and lithology

- aquifer data

What do I use for that? The Utilities or the borehole manager? If both, are they connected? Because I am only able to imput my surface data in the utilities.

If I am using the borehole manager how is my surface defined?

Sorry for bothering you and thanks for your help. I appreciate it

Josch

Share this post


Link to post
Share on other sites

Josch,

The method you are using to produce your grid surface will dictate how long it will take. More complex calculations will add time. When I generated a map from your data I used triangulation because it was a large set. I just ran Easy Map (which uses triangulation) and it took about 4 minutes.

You can certainly do what you are wanting to do but you will need to think about your resolution of the model. A model space that is 200km2 is vary large. It's certainly fine, it's all about scaling. If you want to use your 50m DEM you would need to make your model in 50m spacing and that would be much too large. Depending on the spacing of your boreholes, I would consider a smaller number like 500m grid spacing. I say to go this large because it's a good way to make a faster model that you can get a sense of what's happening before you go into the detailed and more time consuming models at smaller increments.

Use the Borehole manager for storing all of your data except for the grids. It's much more powerful than using the DataSheet. The datasheet has many uses, but the borehole manager will allow you do do things much more efficiently. There is a very 'loose' connection between the BH Manager and the DataSheet. The DataSheet doesn't save to the BH manager except for Stratigraphy and Locations ad those are manual commands. You will typically go back to do things in the Utilities but the Borehole manager is where you want to store all the XYZ data for your boreholes.

Tips on your DEM:

you can use the 50M DEM for cross sections and profiles for the surface, but for doing anything that involves grid math, the surface grid needs to be exactly the same size and dimensions as your model. So if you were going to make a 500m model, make a 500m spacing DEM. While you're at it, make some other resampled ones, maybe at 250, 200, and 100. You can use those later on when you refine your model.

Brian

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×